Mercurial > hgrepos > Python > libs > pygments-lexer-pseudocode2

annotate docs/filters.rst @ 299:11f4f4d2ec03 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Update to latest Sphinx customizations
author Franz Glasner <fzglas.hg@dom66.de>
date Sat, 06 Jun 2026 12:12:06 +0200
parents a094305c5708
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 .. -*- coding: utf-8; indent-tabs-mode: nil; -*-
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2
269
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
3 .. _filters:
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
4
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 *********
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 Filters
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7 *********
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8
265
e5ea2f955986 Optimize documentation layout: use a local ToC where appropriate (lexers, filters)
Franz Glasner <fzglas.hg@dom66.de>
parents: 258
diff changeset
9 .. only:: html
e5ea2f955986 Optimize documentation layout: use a local ToC where appropriate (lexers, filters)
Franz Glasner <fzglas.hg@dom66.de>
parents: 258
diff changeset
10
269
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
11 .. contents::
265
e5ea2f955986 Optimize documentation layout: use a local ToC where appropriate (lexers, filters)
Franz Glasner <fzglas.hg@dom66.de>
parents: 258
diff changeset
12
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
13 The package contains the following filters:
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
14
190
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
15 .. raw:: latex
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
16
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
17 {\footnotesize
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
18
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
19 .. include:: filterlist.rst
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20
190
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
21 .. raw:: latex
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
22
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
23 }
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
24
294
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
25 The :ref:`AlgPseudocode lexer <lexer-algpseudocode>` by default yields an
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
26 error token for the code block below because it encounters an unknown command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
27 When used within `Sphinx`_ it warns about this
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
28 and---as a consequence---suppresses highlighting for this code block
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
29 completely (see also :ref:`this note <note-raiseonerror-filter>`):
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
30
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
31 .. code-block:: none
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
32
206
4e4c83a87387 Change the error token filter examples: use a non-existing command
Franz Glasner <fzglas.hg@dom66.de>
parents: 190
diff changeset
33 \nonexisting{TEST}
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
34
294
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
35 This may be changed by using a custom AlgPseudocode lexer that has
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
36 the ``prohibit_raiseonerror_filter`` lexer option enabled.
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
37 Then the the output in `Sphinx`_ is as follows:
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
38
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
39 .. code-block:: no-raiseonerror-algpseudocode
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
40
206
4e4c83a87387 Change the error token filter examples: use a non-existing command
Franz Glasner <fzglas.hg@dom66.de>
parents: 190
diff changeset
41 \nonexisting{TEST}
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
42
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
43
294
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
44 Alternatively---with the "errortogenericerror" filter applied the very same
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
45 block is highlighted as:
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
46
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
47 .. code-block:: genericerror-algpseudocode
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
48
206
4e4c83a87387 Change the error token filter examples: use a non-existing command
Franz Glasner <fzglas.hg@dom66.de>
parents: 190
diff changeset
49 \nonexisting{TEST}
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
50
294
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
51 The above custom lexer is to be defined in `Sphinx`_ using:
269
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
52
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
53 .. code-block:: python
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
54
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
55 import functools
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
56 from pygments_lexer_pseudocode2.filters import ErrorToGenericErrorTokenFilter
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
57 from pygments_lexer_pseudocode2.lexers.algpseudocode import AlgPseudocodeLexer
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
58
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
59 def setup(app):
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
60 app.add_lexer(
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
61 "genericerror-algpseudocode",
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
62 functools.partial(
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
63 AlgPseudocodeLexer,
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
64 filters=[ErrorToGenericErrorTokenFilter]))
9790553d7225 More details on custom filters in the documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 267
diff changeset
65
294
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
66
278
315e94bac8a7 Link from "Names and Entities" to "TokenReplaceFilter"
Franz Glasner <fzglas.hg@dom66.de>
parents: 270
diff changeset
67 .. _tokenreplacefilter:
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
68
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
69 TokenReplaceFilter
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
70 ==================
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
71
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
72 :Name: tokenreplace
258
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
73 :Filter Options:
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
74 ``replacements``
287
f506d752e801 FIX: the type of a is pygments.token._TokenType -- not pygments.token.Token
Franz Glasner <fzglas.hg@dom66.de>
parents: 278
diff changeset
75 **Type:** :py:class:`dict[str | pygments.token._TokenType, str | pygments.token._TokenType]`
258
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
76
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
77 A map from tokens to their replacements.
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
78
180
63b52570795f FIX: Consistent rendering of lexer or filter options
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
79 ``token_from``
287
f506d752e801 FIX: the type of a is pygments.token._TokenType -- not pygments.token.Token
Franz Glasner <fzglas.hg@dom66.de>
parents: 278
diff changeset
80 **Type:** :py:class:`str` or :py:class:`pygments.token._TokenType`
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
81
172
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
82 The name of a token type (like ``Error``) or a token object
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
83 (like :py:class:`pygments.token.Token.Error`).
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
84
258
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
85 If given the `token_to` options is required and `replacements` will be
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
86 augmented with their respective values.
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
87
180
63b52570795f FIX: Consistent rendering of lexer or filter options
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
88 ``token_to``
287
f506d752e801 FIX: the type of a is pygments.token._TokenType -- not pygments.token.Token
Franz Glasner <fzglas.hg@dom66.de>
parents: 278
diff changeset
89 **Type:** :py:class:`str` or :py:class:`pygments.token._TokenType`
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
90
172
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
91 The name of a token type (like ``Generic.Error``) or a token object
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
92 (like :py:class:`pygments.token.Token.Generic.Error`).
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
93
258
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
94 This option is required if `token_from` is given.
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
95
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
96 Replace all token types given as `replacements` keys or in `token_from`
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
97 with the token types given in `replacements` values or in `token_to`.
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
98
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
99 The values in the token stream are retained.
270
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
100
294
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
101 All :py:class:`str` types in any of the filter options are converted to
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
102 real tokens using `Pygments`_ function
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
103 :py:func:`pygments.token.string_to_tokentype`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 287
diff changeset
104
270
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
105
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
106 ErrorToGenericErrorTokenFilter
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
107 ==============================
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
108
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
109 :Name: errortogenericerror
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
110 :Filter Options: none
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
111
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
112 Replace all :py:class:`pygments.token.Token.Error` tokens in a stream by
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
113 :py:class:`pygments.token.Token.Generic.Error` tokens.
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
114
cb66de44a7b9 Make the filter order more logical.
Franz Glasner <fzglas.hg@dom66.de>
parents: 269
diff changeset
115 The filter is implemented as an application of the `TokenReplaceFilter`_.