Mercurial > hgrepos > Python > libs > pygments-lexer-pseudocode2

annotate tests/test_algpseudo.py @ 289:6fc7f9c1d89d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Remove the old implementation of explicit token types completely
author Franz Glasner <fzglas.hg@dom66.de>
date Fri, 22 May 2026 12:41:08 +0200
parents 298841bc4dee
children b09873e4df58
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 # SPDX-FileCopyrightText: © 2026 Franz Glasner
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 # SPDX-License-Identifier: MIT
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6
53
39151225fb84 Rename the new pseudocode implementation to AlgPseudocode.
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
7 from _tsetup import ALGLEXERFILENAME, ALGLEXERCLASS
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9 import unittest
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
10
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
11 import pygments
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
12 import pygments.lexers
66
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
13 import pygments.formatters
286
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
14 from pygments.token import Token
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
15
288
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
16 from pygments_lexer_pseudocode2.utils import string_to_defined_tokentype
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
17
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
18 import _testhelper
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
19
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
21 class TestSnippets(unittest.TestCase, _testhelper.TokenAssertHelper):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
22
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
23 def setUp(self):
286
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
24 # The standard lexer
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
25 self.lexer = pygments.lexers.load_lexer_from_file(
53
39151225fb84 Rename the new pseudocode implementation to AlgPseudocode.
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
26 ALGLEXERFILENAME, ALGLEXERCLASS)
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
27
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
28 def test_lf(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
29 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
30 [("Text.Whitespace", "\n")],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
31 pygments.lex("\n", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
32
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
33 def test_protected_lf(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
34 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
35 [("Text", "\\\n")],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
36 pygments.lex("\\\n", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
37
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
38 def test_number_int(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
39 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
40 [("Number.Integer", "10"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
41 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
42 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
43 pygments.lex("10", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
44
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
45 def test_number_float_1(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
46 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
47 [("Number.Float", "3.1415926"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
48 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
49 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
50 pygments.lex("3.1415926", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
51
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
52 def test_number_float_2(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
53 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
54 [("Number.Float", "3.14e-12"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
55 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
56 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
57 pygments.lex("3.14e-12", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
58
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
59 def test_string_s_1(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
60 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
61 [("String.Single", "'"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
62 ("String.Single", "HU"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
63 ("String.Single", '"'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
64 ("String.Single", "HE HA"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
65 ("String.Escape", "\\'"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
66 ("String.Single", "HO"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
67 ("String.Single", "'"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
68 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
69 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
70 pygments.lex("""'HU"HE HA\\'HO'""", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
71
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
72 def test_string_s_2(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
73 self.assertTokenStreamEqual(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
74 [("String.Single", "'"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
75 ("String.Single", "HUHU"),
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 41
diff changeset
76 ("Error", "\n"),
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
77 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
78 pygments.lex("'HUHU\nHEHE'", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
79
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
80 def test_string_ts_1(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
81 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
82 [("String.Single", "'''"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
83 ("String.Single", "HUHU HEHE"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
84 ("String.Single", "'''"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
85 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
86 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
87 pygments.lex("'''HUHU HEHE'''", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
88
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
89 def test_string_ts_2(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
90 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
91 [("String.Single", "'''"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
92 ("String.Single", "HI"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
93 ("String.Single", "'"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
94 ("String.Single", "HU"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
95 ("String.Single", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
96 ("String.Single", "HE"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
97 ("String.Single", '"'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
98 ("String.Single", "HA"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
99 ("String.Single", "'''"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
100 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
101 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
102 pygments.lex("""'''HI'HU\nHE"HA'''""", self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
103
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
104 def test_string_d_1(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
105 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
106 [("String.Double", '"'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
107 ("String.Double", 'HU'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
108 ("String.Double", "'"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
109 ("String.Double", 'HE HA'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
110 ("String.Escape", '\\"'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
111 ("String.Double", 'HO'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
112 ("String.Double", '"'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
113 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
114 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
115 pygments.lex('''"HU'HE HA\\"HO"''', self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
116
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
117 def test_string_d_2(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
118 self.assertTokenStreamEqual(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
119 [("String.Double", '"'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
120 ("String.Double", "HUHU"),
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 41
diff changeset
121 ("Error", "\n"),
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
122 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
123 pygments.lex('"HUHU\nHEHE"', self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
124
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
125 def test_string_td_1(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
126 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
127 [("String.Double", '"""'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
128 ("String.Double", 'HUHU HAHA'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
129 ("String.Double", '"""'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
130 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
131 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
132 pygments.lex('"""HUHU HAHA"""', self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
133
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
134 def test_string_td_2(self):
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
135 self.assertTokenStreamEqualComplete(
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
136 [("String.Double", '"""'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
137 ("String.Double", 'HU'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
138 ("String.Double", '"'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
139 ("String.Double", "HO"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
140 ("String.Double", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
141 ("String.Double", "HE"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
142 ("String.Double", "'"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
143 ("String.Double", "HA"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
144 ("String.Double", '"""'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
145 ("Text.Whitespace", "\n"),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
146 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
147 pygments.lex('''"""HU"HO\nHE'HA"""''', self.lexer))
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
148
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
149 def test_proc(self):
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
150 self.assertTokenStreamEqualComplete(
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
151 [("Keyword", "PROCEDURE"),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
152 ("Text.Whitespace", " "),
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
153 ("Name.Entity", "the name"),
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
154 ("Text.Whitespace", "\n"),
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
155 ],
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
156 pygments.lex("\\PROC {the name}", self.lexer))
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
157
105
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
158 def test_proc_with_symbols_in_name(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
159 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
160 [("Keyword", "PROCEDURE"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
161 ("Text.Whitespace", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
162 ("Name.Entity", "the name sqrt ! <="),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
163 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
164 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
165 pygments.lex("\\PROC {the name sqrt ! <=}", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
166
79
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
167 def test_proc_with_escape_in_name(self):
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
168 self.assertTokenStreamEqualComplete(
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
169 [("Keyword", "PROCEDURE"),
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
170 ("Text.Whitespace", " "),
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
171 ("Name.Entity", "t"),
162
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
172 ("Generic.Error", "\\"),
79
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
173 ("Name.Entity", "he "),
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
174 ("Name.Entity", "\\"),
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
175 ("Name.Entity", "}"),
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
176 ("Name.Entity", "name"),
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
177 ("Text.Whitespace", "\n"),
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
178 ],
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
179 pygments.lex(r"\PROC {t\he \\\}name}", self.lexer))
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
180
57
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
181 def test_endproc(self):
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
182 self.assertTokenStreamEqualComplete(
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
183 [("Keyword", "END OF PROCEDURE"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
184 ("Text.Whitespace", "\n"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
185 ],
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
186 pygments.lex("\\END-PROCEDURE", self.lexer))
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
187
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
188 def test_endproc_with_entityname(self):
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
189 self.assertTokenStreamEqualComplete(
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
190 [("Keyword", "END OF PROCEDURE"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
191 ("Text.Whitespace", " "),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
192 ("Name.Entity", "the procedure name"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
193 ("Text.Whitespace", "\n"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
194 ],
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
195 pygments.lex("\\ENDPROCEDURE {the procedure name}", self.lexer))
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
196
94
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
197 def test_endproc_with_entityname_2(self):
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
198 self.assertTokenStreamEqualComplete(
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
199 [("Keyword", "END OF PROCEDURE"),
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
200 ("Text.Whitespace", " "),
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
201 ("Name.Entity", "the procedure name"),
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
202 ("Text.Whitespace", "\n"),
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
203 ],
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
204 pygments.lex("\\END-PROC {the procedure name}", self.lexer))
dc17b778a52b Refactor \END variants
Franz Glasner <fzglas.hg@dom66.de>
parents: 92
diff changeset
205
105
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
206 def test_endproc_with_entityname_3(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
207 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
208 [("Keyword", "END OF PROCEDURE"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
209 ("Text.Whitespace", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
210 ("Name.Entity", "the procedure name with pow and symbols ! <= "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
211 ("Name.Entity", "}"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
212 ("Name.Entity", "<-"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
213 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
214 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
215 pygments.lex(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
216 "\\END-PROC"
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
217 " {the procedure name with pow and symbols ! <= \\}<-}",
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
218 self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
219
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
220 def test_proc_de(self):
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
221 lexer = pygments.lexers.load_lexer_from_file(
53
39151225fb84 Rename the new pseudocode implementation to AlgPseudocode.
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
222 ALGLEXERFILENAME, "AlgPseudocodeLexer_DE")
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
223 self.assertTokenStreamEqualComplete(
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
224 [("Keyword", "PROZEDUR"),
69
b3dd247b27a5 FIX: Add forgotten tests for the new normalized highlighting
Franz Glasner <fzglas.hg@dom66.de>
parents: 66
diff changeset
225 ("Text.Whitespace", " "),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
226 ("Name.Entity", " also {nichtxs"),
74
c1357674622d Remove an escaping backslash where appropriate
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
227 ("Name.Entity", "}"),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
228 ("Name.Entity", " hier"),
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
229 ("Text.Whitespace", "\n"),
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
230 ],
79
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
231 pygments.lex(r"\PROC { also {nichtxs\} hier}", lexer))
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
232
112
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
233 def test_loop_de_not_yet_translated(self):
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
234 lexer = pygments.lexers.load_lexer_from_file(
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
235 ALGLEXERFILENAME, "AlgPseudocodeLexer_DE")
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
236 self.assertTokenStreamEqualComplete(
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
237 [("Keyword", "LOOP"),
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
238 ("Text", " "),
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
239 ("Name.Entity", "a"),
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
240 ("Text.Whitespace", "\n"),
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
241 ],
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
242 pygments.lex(r"\LOOP a", lexer))
ec8767cc5493 More translations (FR and DE)
Franz Glasner <fzglas.hg@dom66.de>
parents: 108
diff changeset
243
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
244 def test_function_1(self):
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
245 self.assertTokenStreamEqualComplete(
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
246 [("Keyword", "FUNCTION"),
69
b3dd247b27a5 FIX: Add forgotten tests for the new normalized highlighting
Franz Glasner <fzglas.hg@dom66.de>
parents: 66
diff changeset
247 ("Text.Whitespace", " "),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
248 ("Name.Entity", "1"),
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
249 ("Text.Whitespace", "\n"),
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
250 ],
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
251 pygments.lex("\\FUNC{1}", self.lexer))
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
252
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
253 def test_function_2(self):
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
254 self.assertTokenStreamEqualComplete(
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
255 [("Keyword", "FUNCTION"),
69
b3dd247b27a5 FIX: Add forgotten tests for the new normalized highlighting
Franz Glasner <fzglas.hg@dom66.de>
parents: 66
diff changeset
256 ("Text.Whitespace", " "),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
257 ("Name.Entity", "line 1\nline 2\n"),
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
258 ("Text.Whitespace", "\n"),
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
259 ],
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
260 pygments.lex("\\FUNCTION{line 1\nline 2\n}", self.lexer))
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
261
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
262 def test_function_3(self):
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
263 self.assertTokenStreamEqualComplete(
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
264 [("Keyword", "FUNCTION"),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
265 ("Text.Whitespace", " "),
162
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
266 ("Generic.Error", "\\"),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
267 ("Name.Entity", "n"),
74
c1357674622d Remove an escaping backslash where appropriate
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
268 ("Name.Entity", "}"),
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
269 ("Text.Whitespace", "\n"),
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
270 ],
79
c14e5fd4d193 More tests with escape characters
Franz Glasner <fzglas.hg@dom66.de>
parents: 78
diff changeset
271 pygments.lex(r"\FN {\n\}}", self.lexer))
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
272
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
273 def test_class(self):
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
274 self.assertTokenStreamEqualComplete(
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
275 [("Keyword", "CLASS"),
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
276 ("Text.Whitespace", " "),
162
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
277 ("Generic.Error", "\\"),
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
278 ("Name.Entity", "n"),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
279 ("Name.Entity", "\\"),
74
c1357674622d Remove an escaping backslash where appropriate
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
280 ("Name.Entity", "}"),
40
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
281 ("Text.Whitespace", "\n"),
df08226a6984 Names for some entities (program, algorithm, function, procedure, class).
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
282 ],
162
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
283 pygments.lex(r"\CLASS {\n\\\}}", self.lexer))
35
d9a3551a1038 Basics of translating some keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
284
41
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
285 def test_class_fr(self):
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
286 lexer = pygments.lexers.load_lexer_from_file(
53
39151225fb84 Rename the new pseudocode implementation to AlgPseudocode.
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
287 ALGLEXERFILENAME, "AlgPseudocodeLexer_FR")
41
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
288 self.assertTokenStreamEqualComplete(
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
289 [("Keyword", "CLASSE"),
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
290 ("Text.Whitespace", " "),
162
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
291 ("Generic.Error", "\\"),
41
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
292 ("Name.Entity", "n"),
74
c1357674622d Remove an escaping backslash where appropriate
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
293 ("Name.Entity", "}"),
41
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
294 ("Text.Whitespace", "\n"),
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
295 ],
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
296 pygments.lex("\\CLASS {\\n\\}}", lexer))
4ccf9a8d0bf2 For the german and french PseudocodeLexer: Use en translations as fallback.
Franz Glasner <fzglas.hg@dom66.de>
parents: 40
diff changeset
297
57
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
298 def test_class_de_with_noend_option(self):
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
299 lexer = pygments.lexers.load_lexer_from_file(
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
300 ALGLEXERFILENAME, "AlgPseudocodeLexer_DE", no_end="True")
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
301 self.assertTokenStreamEqualComplete(
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
302 [("Keyword", "KLASSE"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
303 ("Text.Whitespace", " "),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
304 ("Name.Entity", "\\"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
305 ("Name.Entity", "n"),
74
c1357674622d Remove an escaping backslash where appropriate
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
306 ("Name.Entity", "}"),
57
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
307 ("Text.Whitespace", "\n"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
308 ("Text.Whitespace", "\n"),
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
309 ],
162
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
310 pygments.lex("\\CLASS {\\\\n\\}}\n\\ENDCLASS", lexer))
57
e8f4af9e20a8 Some "ENDxxx" commands
Franz Glasner <fzglas.hg@dom66.de>
parents: 53
diff changeset
311
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
312 def test_class_de_with_noend_option_and_name(self):
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
313 lexer = pygments.lexers.load_lexer_from_file(
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
314 ALGLEXERFILENAME, "AlgPseudocodeLexer_DE", no_end="True")
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
315 self.assertTokenStreamEqualComplete(
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
316 [("Keyword", "KLASSE"),
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
317 ("Text.Whitespace", " "),
162
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
318 ("Generic.Error", "\\"),
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
319 ("Name.Entity", "n"),
74
c1357674622d Remove an escaping backslash where appropriate
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
320 ("Name.Entity", "}"),
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
321 ("Text.Whitespace", "\n"),
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
322 ("Text.Whitespace", "\n"),
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
323 ],
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
324 pygments.lex("\\CLASS {\\n\\}}\n\\ENDCLASS {end class}", lexer))
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
325
39
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
326 def test_remark_1(self):
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
327 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
328 [("Comment.Single", u"▷"),
39
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
329 ("Comment.Single", " the remark"),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
330 ("Text.Whitespace", "\n"),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
331 ],
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
332 pygments.lex("\\REMARK the remark\n", self.lexer))
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
333
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
334 def test_remark_2(self):
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
335 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
336 [("Comment.Single", u"▷"),
39
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
337 ("Comment.Single", " the remark 1"),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
338 ("Text.Whitespace", "\n"),
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
339 ("Comment.Single", u"▷"),
39
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
340 ("Comment.Single", " the remark 2"),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
341 ("Text.Whitespace", "\n"),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
342 ],
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
343 pygments.lex(
64
05c53e431c88 Simplify case-handling for "BLOCK" and "REMARK"
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
344 """\\remArk the remark 1
39
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
345 \\Rem the remark 2
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
346 """, self.lexer))
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 35
diff changeset
347
117
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
348 def test_remark_custom(self):
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
349 # With the APL comment symbol
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
350 lexer = pygments.lexers.load_lexer_from_file(
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
351 ALGLEXERFILENAME, ALGLEXERCLASS, remark=u"⍝")
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
352 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
353 [("Comment.Single", u"⍝"), # U+235D ⍝ APL FUNC. SYMBOL UP SHOE JOT
117
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
354 ("Comment.Single", " another remark"),
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
355 ("Text.Whitespace", "\n"),
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
356 ],
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
357 pygments.lex(u"\\REMARK another remark\n", lexer))
117
d84f1fd10e64 Allow to customize the symbol for a remark
Franz Glasner <fzglas.hg@dom66.de>
parents: 116
diff changeset
358
108
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
359 def test_remark_in_text(self):
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
360 self.assertTokenStreamEqualComplete(
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
361 [("Text", "the text "),
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
362 ("Comment.Single", u"▷"),
108
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
363 ("Comment.Single", " the remark"),
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
364 ("Text.Whitespace", "\n"),
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
365 ("Text", "the next text line"),
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
366 ("Text.Whitespace", "\n"),
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
367 ],
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
368 pygments.lex(
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
369 """\\TEXT{the text \\rem the remark\nthe next text line}""",
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
370 self.lexer))
6cebd3e7bc97 Also allow \REM within a \TEXT{}
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
371
281
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
372 def test_text_short_alias(self):
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
373 self.assertTokenStreamEqualComplete(
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
374 [("Text", "the text "),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
375 ("Text", "nested"),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
376 ("Text", " trailer"),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
377 ("Text.Whitespace", "\n"),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
378 ],
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
379 pygments.lex(
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
380 """\\T{the text \\t{nested} trailer}""",
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
381 self.lexer))
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
382
101
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
383 def test_comment_single_1(self):
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
384 self.assertTokenStreamEqualComplete(
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
385 [("Comment.Single", "// foo bar"),
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
386 ("Text.Whitespace", "\n"),
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
387 ],
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
388 pygments.lex(r"// foo bar", self.lexer))
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
389
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
390 def test_comment_single_2(self):
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
391 self.assertTokenStreamEqualComplete(
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
392 [("Comment.Single", "# foo bar"),
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
393 ("Text.Whitespace", "\n"),
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
394 ],
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
395 pygments.lex(r"# foo bar", self.lexer))
aae16e3624e2 Tests for single-line comments (// and #)
Franz Glasner <fzglas.hg@dom66.de>
parents: 97
diff changeset
396
70
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
397 def test_expr_and_text(self):
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
398 self.assertTokenStreamEqualComplete(
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
399 [("Name.Entity", "a"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
400 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
401 ("Text", "multiplied by"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
402 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
403 ("Name.Entity", "b"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
404 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
405 ("Text", "is"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
406 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
407 ("Number.Integer", "0"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
408 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
409 ("Text", "mod"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
410 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
411 ("Name.Entity", "p"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
412 ("Text.Whitespace", "\n"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
413 ],
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
414 pygments.lex(
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
415 r"""a \TEXT{multiplied by} b \text{is} 0 \text{mod} p""",
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
416 self.lexer))
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
417
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
418 def test_expr_and_text_with_escaped_characters(self):
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
419 self.assertTokenStreamEqualComplete(
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
420 [("Name.Entity", "a"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
421 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
422 ("Text", "multiplied by {escaped"),
72
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
423 ("Text", "}"),
70
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
424 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
425 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
426 ("Name.Entity", "b"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
427 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
428 ("Text", "is "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
429 ("Text", "\\"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
430 ("Text", " not"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
431 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
432 ("Number.Integer", "0"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
433 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
434 ("Text", "mod"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
435 ("Text", " "),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
436 ("Name.Entity", "p"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
437 ("Text.Whitespace", "\n"),
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
438 ],
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
439 pygments.lex(
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
440 r"""a \TEXT{multiplied by {escaped\} } b \text{is \ not} 0"""
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
441 r" \text{mod} p""",
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
442 self.lexer))
5517b0be67f0 Basic "\TEXT" escape for expressions
Franz Glasner <fzglas.hg@dom66.de>
parents: 69
diff changeset
443
75
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
444 def test_punctuation(self):
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
445 self.assertTokenStreamEqualComplete(
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
446 [("Punctuation", "{"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
447 ("Punctuation", "}"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
448 ("Punctuation", ":"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
449 ("Punctuation", "("),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
450 ("Punctuation", ")"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
451 ("Punctuation", ","),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
452 ("Punctuation", ";"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
453 ("Punctuation", "["),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
454 ("Punctuation", "]"),
105
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
455 ("Punctuation", "?"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
456 ("Punctuation", "@"),
75
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
457 ("Text.Whitespace", "\n"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
458 ],
105
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
459 pygments.lex(r"{}:(),;[]?@", self.lexer))
75
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
460
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
461 def test_block_empty(self):
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
462 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
463 [("Text", u"◆"),
75
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
464 ("Text.Whitespace", " "),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
465 ("Text.Whitespace", "\n"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
466 ],
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
467 pygments.lex(r"\block{}", self.lexer))
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
468
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
469 def test_block_with_text(self):
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
470 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
471 [("Text", u"◆"),
75
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
472 ("Text.Whitespace", " "),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
473 ("Text", "a b c"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
474 ("Text.Whitespace", "\n"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
475 ],
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
476 pygments.lex(r"\block{\text{a b c}}", self.lexer))
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
477
76
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
478 def test_block_with_escaped_text(self):
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
479 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
480 [("Text", u"◆"),
76
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
481 ("Text.Whitespace", " "),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
482 ("Text", "\\"),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
483 ("Name.Entity", "text"),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
484 ("Punctuation", "{"),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
485 ("Name.Entity", "a"),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
486 ("Text", " "),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
487 ("Name.Entity", "b"),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
488 ("Text", " "),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
489 ("Name.Entity", "c"),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
490 ("Punctuation", "}"),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
491 ("Text.Whitespace", "\n"),
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
492 ],
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
493 pygments.lex(r"\block{\\text{a b c\}}", self.lexer))
27e12387154d FIX: Handle escape character more thoroughly: also explicitely handle "\\"
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
494
75
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
495 def test_block(self):
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
496 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
497 [("Text", u"◆"),
75
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
498 ("Text.Whitespace", " "),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
499 ("Name.Entity", "a"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
500 ("Text", " "),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
501 ("Number.Float", "1.2"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
502 ("Text", " "),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
503 ("Punctuation", "{"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
504 ("Name.Entity", "x"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
505 ("Text", " "),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
506 ("Operator.Word", "in"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
507 ("Text", " "),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
508 ("Name.Entity", "X"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
509 ("Punctuation", "}"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
510 ("Text", " "),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
511 ("Name.Entity", "c"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
512 ("Text.Whitespace", "\n"),
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
513 ],
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
514 pygments.lex(r"\state{a 1.2 {x in X\} c}", self.lexer))
711f8d19e27a New implementation of "STATEMENT" (also aliased to "STATE" and "BLOCK").
Franz Glasner <fzglas.hg@dom66.de>
parents: 74
diff changeset
515
78
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
516 def test_tstate_empty(self):
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
517 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
518 [("Text", u"▪"),
78
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
519 ("Text.Whitespace", " "),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
520 ("Text.Whitespace", "\n"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
521 ],
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
522 pygments.lex(r"\tstate{}", self.lexer))
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
523
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
524 def test_tstate_with_expr(self):
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
525 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
526 [("Text", u"▪"),
78
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
527 ("Text.Whitespace", " "),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
528 ("Text", "a 1.2 "),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
529 ("Name.Entity", "x"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
530 ("Text", " "),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
531 ("Operator.Word", "in"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
532 ("Text", " "),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
533 ("Name.Entity", "X"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
534 ("Punctuation", "}"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
535 ("Text", " c"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
536 ("Text.Whitespace", "\n"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
537 ],
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
538 pygments.lex(r"\tstate{a 1.2 \expr{x in X\}} c}", self.lexer))
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
539
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
540 def test_tstate_with_escaped_expr(self):
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
541 self.assertTokenStreamEqualComplete(
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
542 [("Text", u"▪"),
78
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
543 ("Text.Whitespace", " "),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
544 ("Text", "a 1.2 "),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
545 ("Text", "\\"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
546 ("Text", "expr{x in X"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
547 ("Text", "}"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
548 ("Text", " c"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
549 ("Text.Whitespace", "\n"),
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
550 ],
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
551 pygments.lex(r"\tstate{a 1.2 \\expr{x in X\} c}", self.lexer))
abb2e8c65d0f Implement "TEXTSTATEMENT"
Franz Glasner <fzglas.hg@dom66.de>
parents: 76
diff changeset
552
281
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
553 def test_expr_short_alias(self):
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
554 self.assertTokenStreamEqualComplete(
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
555 [("Name.Entity", "foo"),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
556 ("Text", " "),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
557 ("Name.Entity", "bar"),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
558 ("Text", " "),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
559 ("Text", "bar +"),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
560 ("Text.Whitespace", "\n"),
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
561 ],
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
562 pygments.lex(r"foo \e{bar} \t{bar +}", self.lexer))
ee512932d603 Make "\T" and "\E" aliases for "\TEXT" and "\EXPRESSION"
Franz Glasner <fzglas.hg@dom66.de>
parents: 276
diff changeset
563
92
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
564 def test_text_in_expr(self):
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
565 self.assertTokenStreamEqualComplete(
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
566 [("Name.Entity", "first"),
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
567 ("Text", " "),
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
568 ("Name.Entity", "expression"),
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
569 ("Text", " "),
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
570 ("Name.Entity", "second"),
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
571 ("Text", " "),
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
572 ("Name.Entity", "expression"),
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
573 ("Text.Whitespace", "\n"),
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
574 ],
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
575 pygments.lex(
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
576 r"\TEXT{\EXPRESSION{first expression} "
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
577 r"\EXPR{second expression}}",
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
578 self.lexer))
3f37832c405d FIX: Wrong state for \TEXT command
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
579
285
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
580 def test_extended_explicit_tokentype_empty(self):
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
581 self.assertTokenStreamEqualComplete(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
582 [("Text.Whitespace", "\n")],
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
583 pygments.lex(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
584 r"""\ttX{nd}{}""", self.lexer))
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
585
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
586 def test_extended_explicit_tokentype_simple(self):
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
587 self.assertTokenStreamEqualComplete(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
588 [("Name.Decorator", "simple\nline 2"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
589 ("Text.Whitespace", "\n"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
590 ],
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
591 pygments.lex(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
592 """\\ttX{nd}{simple\nline 2}""", self.lexer))
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
593
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
594 def test_extended_explicit_tokentype_escaped_brace(self):
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
595 self.assertTokenStreamEqualComplete(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
596 [("Name.Decorator", "simple"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
597 ("Name.Decorator", "}"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
598 ("Name.Decorator", "part 2"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
599 ("Text.Whitespace", "\n"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
600 ],
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
601 pygments.lex(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
602 """\\ttX{nd}{simple\\}part 2}""", self.lexer))
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
603
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
604 def test_extended_explicit_tokentype_escaped_backslash(self):
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
605 self.assertTokenStreamEqualComplete(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
606 [("Name.Decorator", "simple"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
607 ("Name.Decorator", "\\"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
608 ("Name.Decorator", "part 2"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
609 ("Text.Whitespace", "\n"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
610 ],
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
611 pygments.lex(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
612 """\\ttX{nd}{simple\\\\part 2}""", self.lexer))
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
613
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
614 def test_extended_explicit_tokentype_single_backslash(self):
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
615 self.assertTokenStreamEqualComplete(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
616 [("Name.Variable", "simple"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
617 ("Generic.Error", "\\"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
618 ("Name.Variable", "part 2"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
619 ("Text.Whitespace", "\n"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
620 ],
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
621 pygments.lex(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
622 """\\ttX{nv}{simple\\part 2}""", self.lexer))
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
623
286
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
624 def test_extended_explicit_tokentype_non_existing_type_strict(self):
285
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
625 self.assertTokenStreamEqualComplete(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
626 [("Generic.Error", u"\\ttX{NON_EXISTING}{"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
627 ("Generic.Error", u"∈_∌"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
628 ("Generic.Error", u"}"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
629 ("Punctuation", "("),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
630 ("Name.Entity", "p1"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
631 ("Punctuation", ","),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
632 ("Text", " "),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
633 ("Name.Entity", "p2"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
634 ("Punctuation", ")"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
635 ("Text.Whitespace", "\n"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
636 ],
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
637 pygments.lex(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
638 u"\\ttX{NON_EXISTING}{∈_∌}(p1, p2)", self.lexer))
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
639
286
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
640 def test_extended_explicit_tokentype_non_existing_type_lenient(self):
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
641 lexer = pygments.lexers.load_lexer_from_file(
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
642 ALGLEXERFILENAME, ALGLEXERCLASS, strict_tokentype=False)
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
643 self.assertTokenStreamEqualComplete(
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
644 [(getattr(getattr(getattr(Token, "DOES"), "IT"), "NOT-EXIST"),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
645 u"∈_∌"),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
646 ("Punctuation", "("),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
647 ("Name.Entity", "p1"),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
648 ("Punctuation", ","),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
649 ("Text", " "),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
650 ("Name.Entity", "p2"),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
651 ("Punctuation", ")"),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
652 ("Text.Whitespace", "\n"),
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
653 ],
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
654 pygments.lex(
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
655 u"\\ttX{DOES.IT.NOT-EXIST}{∈_∌}(p1, p2)", lexer))
051c8877ee22 Implement lexer option "strict_tokentype".
Franz Glasner <fzglas.hg@dom66.de>
parents: 285
diff changeset
656
285
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
657 def test_just_braces_in_expressions(self):
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
658 self.assertTokenStreamEqualComplete(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
659 [("Punctuation", "{"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
660 ("Name.Entity", "foo"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
661 ("Text", " "),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
662 ("Name.Entity", "bar"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
663 ("Punctuation", "}"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
664 ("Text.Whitespace", "\n"),
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
665 ],
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
666 pygments.lex(
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
667 r"{foo bar}", self.lexer))
afbca50b7dc1 Implement an alternate syntax for "Explicit Token Types".
Franz Glasner <fzglas.hg@dom66.de>
parents: 281
diff changeset
668
97
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
669 def test_end_combinations(self):
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
670 self.assertTokenStreamEqualComplete(
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
671 [("Keyword", "BEGIN"),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
672 ("Text", " "),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
673 ("Keyword", "END"),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
674 ("Text", " "),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
675 ("Keyword", "END OF FUNCTION"),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
676 ("Text", " "),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
677 ("Keyword", "END OF FUNCTION"),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
678 ("Text.Whitespace", " "),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
679 ("Name.Entity", "An End of a Function"),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
680 ("Text", " "),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
681 ("Keyword", "END OF FUNCTION"),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
682 ("Text.Whitespace", " "),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
683 ("Name.Entity", "The End of the Next Function"),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
684 ("Text", " "),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
685 ("Text.Whitespace", "\n"),
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
686 ],
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
687 pygments.lex(
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
688 r"""\begin \end \end-fn \end-fn {An End of a Function}"""
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
689 r""" \end fn {The End of the Next Function} """,
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
690 self.lexer))
dd4eb937485c A couple of new keywords
Franz Glasner <fzglas.hg@dom66.de>
parents: 95
diff changeset
691
105
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
692 def test_unicode_math_operator(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
693 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
694 [("Operator", u"∈"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
695 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
696 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
697 pygments.lex(u"∈", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
698
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
699 def test_ascii_math_operator(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
700 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
701 [("Operator", "="),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
702 ("Operator", "!"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
703 ("Operator", "&"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
704 ("Operator", "<"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
705 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
706 ("Operator", ">"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
707 ("Operator", "+"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
708 ("Operator", "-"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
709 ("Operator", "*"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
710 ("Operator", "/"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
711 ("Operator", "%"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
712 ("Operator", "|"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
713 ("Operator", "~"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
714 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
715 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
716 pygments.lex("=!&< >+-*/%|~", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
717
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
718 def test_ascii_math_operator_with_replacements(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
719 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
720 [("Operator", u"⇔"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
721 ("Text", " "),
116
9bfd87544902 Use arrows from Supplemental Arrows-A Unicode block where appropriate: better readability
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
722 ("Operator", u"⟷"),
105
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
723 ("Text", " "),
116
9bfd87544902 Use arrows from Supplemental Arrows-A Unicode block where appropriate: better readability
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
724 ("Operator", u"⟵"),
105
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
725 ("Text", " "),
116
9bfd87544902 Use arrows from Supplemental Arrows-A Unicode block where appropriate: better readability
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
726 ("Operator", u"⟶"),
105
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
727 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
728 ("Operator", u"⇒"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
729 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
730 ("Operator", u"≤"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
731 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
732 ("Operator", u"≥"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
733 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
734 ("Operator", u"≠"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
735 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
736 ("Operator", u"≠"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
737 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
738 ("Operator", u"∶="), # u"≔"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
739 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
740 ("Operator", u"=∶"), # u"≕"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
741 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
742 ("Operator", u"≟"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
743 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
744 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
745 pygments.lex(u"<=> <-> <- -> => <= >= <> != := =: ?=", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
746
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
747 def test_word_operators(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
748 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
749 [("Operator.Word", "IN"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
750 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
751 ("Operator.Word", "is"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
752 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
753 ("Operator.Word", "And"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
754 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
755 ("Operator.Word", "Or"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
756 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
757 ("Operator.Word", "XOR"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
758 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
759 ("Operator.Word", "not"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
760 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
761 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
762 pygments.lex("IN is And Or XOR not", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
763
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
764 def test_keyword_constants(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
765 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
766 [("Keyword.Constant", "true"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
767 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
768 ("Keyword.Constant", "FALSE"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
769 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
770 ("Keyword.Constant", "None"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
771 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
772 ("Keyword.Constant", "nil"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
773 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
774 ("Keyword.Constant", "NULL"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
775 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
776 ("Keyword.Constant", "Empty"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
777 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
778 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
779 pygments.lex("true FALSE None nil NULL Empty", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
780
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
781 def test_math_builtins(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
782 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
783 [("Name.Builtin", "sqrt"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
784 ("Punctuation", "("),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
785 ("Name.Entity", "Foo"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
786 ("Punctuation", ")"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
787 ("Punctuation", ";"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
788 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
789 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
790 pygments.lex("sqrt(Foo);", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
791
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
792 def test_math_builtins_nested(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
793 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
794 [("Name.Builtin", "pow"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
795 ("Punctuation", "("),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
796 ("Number.Integer", "2"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
797 ("Punctuation", ","),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
798 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
799 ("Number.Integer", "8"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
800 ("Punctuation", ")"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
801 ("Punctuation", ";"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
802 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
803 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
804 pygments.lex("\\text{\\expr{pow(2, 8);}}", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
805
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
806 def test_math_letters(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
807 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
808 [("Name.Entity", "a"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
809 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
810 ("Operator.Word", "in"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
811 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
812 ("Name.Entity", u"ℂ"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
813 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
814 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
815 pygments.lex(u"a in ℂ", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
816
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
817 def test_other_symbols(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
818 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
819 [("Name.Entity", "b"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
820 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
821 ("Operator.Word", "in"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
822 ("Text", " "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
823 ("Text", u"℀"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
824 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
825 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
826 pygments.lex(u"b in ℀", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
827
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
828 def test_escaped_string_start_1(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
829 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
830 [("Text", "flow "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
831 ("Name.Entity", "f"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
832 ("Punctuation", "'"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
833 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
834 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
835 pygments.lex(r"\TEXT{flow \expr{f\'}}", self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
836
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
837 def test_escaped_string_start_2(self):
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
838 self.assertTokenStreamEqualComplete(
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
839 [("Text", "flow "),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
840 ("Name.Entity", "f"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
841 ("Punctuation", '"'),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
842 ("Text.Whitespace", "\n"),
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
843 ],
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
844 pygments.lex(r'\TEXT{flow \expr{f\"}}', self.lexer))
cec52d83869a Handle much more characters from the Unicode codeset in expressions.
Franz Glasner <fzglas.hg@dom66.de>
parents: 101
diff changeset
845
113
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
846 def test_gets_default(self):
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
847 self.assertTokenStreamEqualComplete(
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
848 [("Name.Entity", "a"),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
849 ("Text", " "),
123
4d96ace53ba1 Make it work on Python2 too with all tests by explicitely declaring some strings to be Unicode strings.
Franz Glasner <fzglas.hg@dom66.de>
parents: 117
diff changeset
850 ("Operator", u"⟵"), # U+27F5 (Supplemental Arrows-A)
113
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
851 ("Text", " "),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
852 ("Number.Integer", "2"),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
853 ("Text.Whitespace", "\n"),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
854 ],
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
855 pygments.lex(r"a \gets 2", self.lexer))
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
856
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
857 def test_gets_custom(self):
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
858 lexer = pygments.lexers.load_lexer_from_file(
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
859 ALGLEXERFILENAME, ALGLEXERCLASS, gets="::=")
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
860 self.assertTokenStreamEqualComplete(
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
861 [("Name.Entity", "a"),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
862 ("Text", " "),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
863 ("Operator", "::="),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
864 ("Text", " "),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
865 ("Number.Float", "2.7"),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
866 ("Text.Whitespace", "\n"),
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
867 ],
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
868 pygments.lex(r"a \gets 2.7", lexer))
9aea2fd5f037 The \GETS command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 112
diff changeset
869
114
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
870 def test_call(self):
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
871 self.assertTokenStreamEqualComplete(
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
872 [("Name.Entity", "a function"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
873 ("Punctuation", "("),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
874 ("Name.Entity", "p1"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
875 ("Punctuation", ","),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
876 ("Text", " "),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
877 ("Name.Entity", "p2"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
878 ("Punctuation", ")"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
879 ("Text.Whitespace", "\n"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
880 ],
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
881 pygments.lex(r"\CALL{a function}(p1, p2)", self.lexer))
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
882
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
883 def test_call_in_expr(self):
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
884 self.assertTokenStreamEqualComplete(
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
885 [("Name.Entity", "a procedure"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
886 ("Punctuation", "("),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
887 ("Name.Entity", "arg1"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
888 ("Punctuation", ","),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
889 ("Text", " "),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
890 ("Name.Entity", "arg2"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
891 ("Punctuation", ","),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
892 ("Text", " "),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
893 ("Name.Entity", "arg3"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
894 ("Punctuation", ")"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
895 ("Text.Whitespace", "\n"),
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
896 ],
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
897 pygments.lex(r"\TEXT{\EXPR{\CALL{a procedure}(arg1, arg2, arg3)}}",
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
898 self.lexer))
be50fe0687d6 The \CALL command
Franz Glasner <fzglas.hg@dom66.de>
parents: 113
diff changeset
899
149
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
900 def test_name(self):
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
901 self.assertTokenStreamEqualComplete(
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
902 [("Name.Entity", "an entity"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
903 ("Punctuation", "("),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
904 ("Name.Entity", "p1"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
905 ("Punctuation", ","),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
906 ("Text", " "),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
907 ("Name.Entity", "p2"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
908 ("Punctuation", ")"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
909 ("Text.Whitespace", "\n"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
910 ],
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
911 pygments.lex(r"\NAME{an entity}(p1, p2)", self.lexer))
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
912
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
913 def test_name_in_expr(self):
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
914 self.assertTokenStreamEqualComplete(
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
915 [("Name.Entity", "an entity"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
916 ("Punctuation", "("),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
917 ("Name.Entity", "arg1"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
918 ("Punctuation", ","),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
919 ("Text", " "),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
920 ("Name.Entity", "arg2"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
921 ("Punctuation", ","),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
922 ("Text", " "),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
923 ("Name.Entity", "arg3"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
924 ("Punctuation", ")"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
925 ("Text.Whitespace", "\n"),
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
926 ],
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
927 pygments.lex(r"\TEXT{\EXPR{\NAME{an entity}(arg1, arg2, arg3)}}",
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
928 self.lexer))
380d2607d5c7 Implement the "\NAME" command.
Franz Glasner <fzglas.hg@dom66.de>
parents: 123
diff changeset
929
115
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
930 def test_comment_multi(self):
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
931 self.assertTokenStreamEqualComplete(
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
932 [("Comment.Multiline", "/*"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
933 ("Comment.Multiline", " word1 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
934 ("Comment.Multiline", "*"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
935 ("Comment.Multiline", " word2 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
936 ("Comment.Multiline", "/"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
937 ("Comment.Multiline", " word3 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
938 ("Comment.Multiline", "/"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
939 ("Comment.Multiline", "/"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
940 ("Comment.Multiline", " word4 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
941 ("Comment.Multiline", "*/"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
942 ("Text.Whitespace", "\n"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
943 ],
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
944 pygments.lex(r"/* word1 * word2 / word3 // word4 */", self.lexer))
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
945
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
946 def test_comment_multi_alt(self):
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
947 self.assertTokenStreamEqualComplete(
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
948 [("Comment.Multiline", "(*"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
949 ("Comment.Multiline", " word1 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
950 ("Comment.Multiline", "*"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
951 ("Comment.Multiline", " word2 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
952 ("Comment.Multiline", "("),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
953 ("Comment.Multiline", " word3 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
954 ("Comment.Multiline", ")"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
955 ("Comment.Multiline", " word4 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
956 ("Comment.Multiline", "("),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
957 ("Comment.Multiline", ")"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
958 ("Comment.Multiline", " word5 "),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
959 ("Comment.Multiline", "*)"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
960 ("Text.Whitespace", "\n"),
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
961 ],
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
962 pygments.lex(r"(* word1 * word2 ( word3 ) word4 () word5 *)",
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
963 self.lexer))
e1663ac707b0 An alternate multiline comment syntax using "(* ... *)".
Franz Glasner <fzglas.hg@dom66.de>
parents: 114
diff changeset
964
159
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
965 def test_dots_one(self):
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
966 self.assertTokenStreamEqualComplete(
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
967 [("Punctuation", "."),
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
968 ("Text.Whitespace", "\n"),
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
969 ],
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
970 pygments.lex(r".", self.lexer))
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
971
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
972 def test_dots_two(self):
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
973 self.assertTokenStreamEqualComplete(
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
974 [("Punctuation", ".."),
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
975 ("Text.Whitespace", "\n"),
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
976 ],
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
977 pygments.lex(r"..", self.lexer))
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
978
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
979 def test_dots_three(self):
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
980 self.assertTokenStreamEqualComplete(
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
981 [("Punctuation", "..."),
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
982 ("Text.Whitespace", "\n"),
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
983 ],
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
984 pygments.lex(r"...", self.lexer))
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
985
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
986 def test_dots_four(self):
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
987 self.assertTokenStreamEqualComplete(
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
988 [("Punctuation", "...."),
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
989 ("Text.Whitespace", "\n"),
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
990 ],
162
11ce0903ff8b Yield lone backslash characters in expressions that not really escape anything as "Generic.Error" now.
Franz Glasner <fzglas.hg@dom66.de>
parents: 159
diff changeset
991 pygments.lex(r"....", self.lexer))
159
4ee0b1536ea6 Handle runs of dots in expressions (., .., ..., ...., ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 149
diff changeset
992
212
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
993 def test_nested_expr(self):
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
994 self.assertTokenStreamEqualComplete(
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
995 [("Name.Entity", "foo"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
996 ("Text", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
997 ("Operator", "+"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
998 ("Text", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
999 ("Name.Entity", "bar"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1000 ("Text", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1001 ("Operator", "*"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1002 ("Text", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1003 ("Number.Integer", '7'),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1004 ("Text.Whitespace", "\n"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1005 ],
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1006 pygments.lex(r"\EXPR{\EXPR{foo + bar} * 7}", self.lexer))
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1007
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1008 def test_nested_expr_in_block(self):
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1009 self.assertTokenStreamEqualComplete(
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1010 [("Text", u"◆"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1011 ("Text.Whitespace", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1012 ("Name.Entity", "foo"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1013 ("Text", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1014 ("Operator", "+"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1015 ("Text", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1016 ("Name.Entity", "bar"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1017 ("Text", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1018 ("Operator", "*"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1019 ("Text", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1020 ("Number.Integer", '7'),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1021 ("Text.Whitespace", "\n"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1022 ],
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1023 pygments.lex(u"\\BLOCK{\\EXPR{\\EXPR{foo + bar} * 7}}",
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1024 self.lexer))
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1025
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1026 def test_nested_text(self):
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1027 self.assertTokenStreamEqualComplete(
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1028 [("Text", "foo + bar"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1029 ("Text", " * 7"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1030 ("Text.Whitespace", "\n"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1031 ],
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1032 pygments.lex(r"\TEXT{\TEXT{foo + bar} * 7}", self.lexer))
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1033
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1034 def test_nested_text_in_tblock(self):
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1035 self.assertTokenStreamEqualComplete(
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1036 [("Text", u"▪"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1037 ("Text.Whitespace", " "),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1038 ("Text", "bar + Foo"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1039 ("Text", " * 7"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1040 ("Text.Whitespace", "\n"),
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1041 ],
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1042 pygments.lex(u"\\TBLOCK{\\TEXT{\\TEXT{bar + Foo} * 7}}",
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1043 self.lexer))
18553f595b34 Allow nested \TEXT and \EXPR commands.
Franz Glasner <fzglas.hg@dom66.de>
parents: 162
diff changeset
1044
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1045
66
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1046 class PygmentizeCompletely(unittest.TestCase):
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1047
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1048 def setUp(self):
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1049 self.lexer = pygments.lexers.load_lexer_from_file(
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1050 ALGLEXERFILENAME, ALGLEXERCLASS)
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1051
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1052 def test_pygmentize_html(self):
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1053 html_formatter = pygments.formatters.get_formatter_by_name("html")
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1054 highlighted = pygments.highlight(
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1055 '"""FOO"""',
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1056 self.lexer,
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1057 html_formatter,
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1058 outfile=None)
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1059 self.assertTrue(highlighted.startswith("""<div class="highlight">"""))
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1060
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1061 def test_pygmentize_latex(self):
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1062 latex_formatter = pygments.formatters.get_formatter_by_name("latex")
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1063 highlighted = pygments.highlight(
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1064 '"""FOO"""',
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1065 self.lexer,
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1066 latex_formatter,
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1067 outfile=None)
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1068 self.assertTrue(highlighted.startswith(r"""\begin{Verbatim}"""))
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1069
9ba1584d4b4a Also add a test that completely highlights a document with Pygments
Franz Glasner <fzglas.hg@dom66.de>
parents: 64
diff changeset
1070
288
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1071 class TokenBehaviour(unittest.TestCase):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1072
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1073 def test_string_to_defined_tokentype(self):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1074 t = string_to_defined_tokentype("Generic")
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1075 self.assertIs(t, Token.Generic)
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1076
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1077 def test_string_to_defined_tokentype_2(self):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1078 t = string_to_defined_tokentype("Generic.Error")
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1079 self.assertIs(t, Token.Generic.Error)
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1080
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1081 def test_string_to_defined_tokentype_empty(self):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1082 t = string_to_defined_tokentype("")
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1083 self.assertIs(t, Token)
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1084
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1085 def test_string_to_undefined_tokentype(self):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1086 self.assertIsNone(string_to_defined_tokentype("non-existing"))
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1087 # twice to assert that it it not created by the call
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1088 self.assertIsNone(string_to_defined_tokentype("non-existing"))
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1089
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1090 def test_string_to_undefined_tokentype_2(self):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1091 self.assertIsNone(string_to_defined_tokentype("Generic.non-existing"))
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1092 # twice to assert that it it not created by the call
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1093 self.assertIsNone(string_to_defined_tokentype("Generic.non-existing"))
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1094
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1095 def test_string_to_undefined_tokentype_3(self):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1096 self.assertIsNone(string_to_defined_tokentype("Not.Yet.Existing"))
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1097 # twice to assert that it it not created by the call
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1098 self.assertIsNone(string_to_defined_tokentype("Not.Yet.Existing"))
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1099
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1100 def test_string_to_undefined_tokentype_4(self):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1101 self.assertIsNone(string_to_defined_tokentype("Generic..Error"))
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1102 # twice to assert that it it not created by the call
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1103 self.assertIsNone(string_to_defined_tokentype("Generic..Error"))
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1104
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 286
diff changeset
1105
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1106 if __name__ == "__main__":
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1107 unittest.main()