annotate docs/filters.rst @ 263:2e78e255b54e

Make a link to the reST source where appropriate to learn about the lexer syntax
author Franz Glasner <fzglas.hg@dom66.de>
date Tue, 19 May 2026 18:03:48 +0200
parents e03e8cfdcece
children e5ea2f955986
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 .. -*- coding: utf-8; indent-tabs-mode: nil; -*-
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 *********
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 Filters
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 *********
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7 The package contains the following filters:
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8
190
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
9 .. raw:: latex
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
10
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
11 {\footnotesize
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
12
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
13 .. include:: filterlist.rst
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
14
190
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
15 .. raw:: latex
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
16
8825282f6900 Select a smaller font size for all lexer and filter tables in the PDF output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 180
diff changeset
17 }
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
18
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
19 The AlgPseudocode lexer yields an error token for the following code block.
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20 `Sphinx`_ therefore suppresses highlighting completely:
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
21
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
22 .. code-block:: none
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
23
206
4e4c83a87387 Change the error token filter examples: use a non-existing command
Franz Glasner <fzglas.hg@dom66.de>
parents: 190
diff changeset
24 \nonexisting{TEST}
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
25
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
26 With a custom AlgPseudocode lexer that has ``prohibit_raiseonerror_filter``
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
27 activated the output in `Sphinx`_ is as:
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
28
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
29 .. code-block:: no-raiseonerror-algpseudocode
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
30
206
4e4c83a87387 Change the error token filter examples: use a non-existing command
Franz Glasner <fzglas.hg@dom66.de>
parents: 190
diff changeset
31 \nonexisting{TEST}
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
32
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
33
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
34 With the "errortogenericerror" filter the very same block is highlighted
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
35 as:
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
36
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
37 .. code-block:: genericerror-algpseudocode
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
38
206
4e4c83a87387 Change the error token filter examples: use a non-existing command
Franz Glasner <fzglas.hg@dom66.de>
parents: 190
diff changeset
39 \nonexisting{TEST}
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
40
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
41
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
42 ErrorToGenericErrorTokenFilter
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
43 ==============================
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
44
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
45 :Name: errortogenericerror
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
46 :Filter Options: none
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
47
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
48 Replace all :py:class:`pygments.token.Token.Error` tokens in a stream by
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
49 :py:class:`pygments.token.Token.Generic.Error` tokens.
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
50
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
51
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
52 TokenReplaceFilter
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
53 ==================
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
54
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
55 :Name: tokenreplace
258
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
56 :Filter Options:
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
57 ``replacements``
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
58 **Type:** :py:class:`dict[str | pygments.token.Token, str | pygments.token.Token]`
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
59
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
60 A map from tokens to their replacements.
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
61
180
63b52570795f FIX: Consistent rendering of lexer or filter options
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
62 ``token_from``
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
63 **Type:** :py:class:`str` or :py:class:`pygments.token.Token`
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
64
172
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
65 The name of a token type (like ``Error``) or a token object
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
66 (like :py:class:`pygments.token.Token.Error`).
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
67
258
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
68 If given the `token_to` options is required and `replacements` will be
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
69 augmented with their respective values.
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
70
180
63b52570795f FIX: Consistent rendering of lexer or filter options
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
71 ``token_to``
168
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
72 **Type:** :py:class:`str` or :py:class:`pygments.token.Token`
bff8b900713a REFACTOR: All documentation pages refactored: merge intro and details for lexers and filters
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
73
172
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
74 The name of a token type (like ``Generic.Error``) or a token object
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
75 (like :py:class:`pygments.token.Token.Generic.Error`).
ad80fcbf7b47 Docs formatting
Franz Glasner <fzglas.hg@dom66.de>
parents: 168
diff changeset
76
258
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
77 This option is required if `token_from` is given.
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
78
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
79 Replace all token types given as `replacements` keys or in `token_from`
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
80 with the token types given in `replacements` values or in `token_to`.
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
81
e03e8cfdcece Make the TokenReplaceFilter more flexible by allowing a replacement map
Franz Glasner <fzglas.hg@dom66.de>
parents: 206
diff changeset
82 The values in the token stream are retained.