129 lines
5.8 KiB
Plaintext
129 lines
5.8 KiB
Plaintext
README for testing lexers with lexilla/test.
|
|
|
|
The TestLexers application is run to test the lexing and folding of a set of example
|
|
files and thus ensure that the lexers are working correctly.
|
|
|
|
Lexers are accessed through the Lexilla shared library which must be built first
|
|
in the lexilla/src directory.
|
|
|
|
TestLexers works on Windows, Linux, or macOS and requires a C++20 compiler.
|
|
MSVC 2019.4, GCC 9.0, Clang 9.0, and Apple Clang 11.0 are known to work.
|
|
|
|
MSVC is only available on Windows.
|
|
|
|
GCC and Clang work on Windows and Linux.
|
|
|
|
On macOS, only Apple Clang is available.
|
|
|
|
Lexilla requires some headers from Scintilla to build and expects a directory named
|
|
"scintilla" containing a copy of Scintilla 5+ to be a peer of the Lexilla top level
|
|
directory conventionally called "lexilla".
|
|
|
|
To use GCC run lexilla/test/makefile:
|
|
make test
|
|
|
|
To use Clang run lexilla/test/makefile:
|
|
make CLANG=1 test
|
|
On macOS, CLANG is set automatically so this can just be
|
|
make test
|
|
|
|
To use MSVC:
|
|
nmake -f testlexers.mak test
|
|
There is also a project file TestLexers.vcxproj that can be loaded into the Visual
|
|
C++ IDE.
|
|
|
|
|
|
|
|
Adding or Changing Tests
|
|
|
|
The lexilla/test/examples directory contains a set of tests located in a tree of
|
|
subdirectories.
|
|
|
|
Each directory contains example files along with control files called
|
|
SciTE.properties and expected result files with .styled and .folded suffixes.
|
|
If an unexpected result occurs then files with the additional suffix .new
|
|
(that is .styled.new or .folded.new) may be created.
|
|
|
|
Each file in the examples tree that does not have an extension of .properties, .styled,
|
|
.folded or .new is an example file that will be lexed and folded according to settings
|
|
found in SciTE.properties.
|
|
|
|
The results of the lex will be compared to the corresponding .styled file and if different
|
|
the result will be saved to a .styled.new file for checking.
|
|
So, if x.cxx is the example, its lexed form will be checked against x.cxx.styled and a
|
|
x.cxx.styled.new file may be created. The .styled.new and .styled files contain the text
|
|
of the original file along with style number changes in {} like:
|
|
{5}function{0} {11}first{10}(){0}
|
|
After checking that the .styled.new file is correct, it can be promoted to .styled and
|
|
committed to the repository.
|
|
|
|
The results of the fold will be compared to the corresponding .folded file and if different
|
|
the result will be saved to a .folded.new file for checking.
|
|
So, if x.cxx is the example, its folded form will be checked against x.cxx.folded and a
|
|
x.cxx.folded.new file may be created. The folded.new and .folded files contain the text
|
|
of the original file along with fold information to the left like:
|
|
|
|
2 400 0 + --[[ coding:UTF-8
|
|
0 402 0 | comment ]]
|
|
|
|
There are 4 columns before the file text representing the bits of the fold level:
|
|
[flags (0xF000), level (0x0FFF), other (0xFFFF0000), picture].
|
|
flags: may be 2 for header or 1 for whitespace.
|
|
level: hexadecimal level number starting at 0x400. 'negative' level numbers like 0x3FF
|
|
indicate errors in either the folder or in the input file, such as a C file that starts with #endif.
|
|
other: can be used as the folder wants. Often used to hold the level of the next line.
|
|
picture: gives a rough idea of the fold structure: '|' for level greater than 0x400,
|
|
'+' for header, ' ' otherwise.
|
|
After checking that the .folded.new file is correct, it can be promoted to .folded and
|
|
committed to the repository.
|
|
|
|
An interactive file comparison program like WinMerge (https://winmerge.org/) on
|
|
Windows or meld (https://meldmerge.org/) on Linux can help examine differences
|
|
between the .styled and .styled.new files or .folded and .folded.new files.
|
|
|
|
On Windows, the scripts/PromoteNew.bat script can be run to promote all .new result
|
|
files to their base names without .new.
|
|
|
|
Styling and folding tests are first performed on the file as a whole, then the file is lexed
|
|
and folded line-by-line. If there are differences between the whole file and line-by-line
|
|
then a message with 'per-line is different' for styling or 'per-line has different folds' will be
|
|
printed. Problems with line-by-line processing are often caused by local variables in the
|
|
lexer or folder that are incorrectly initialised. Sometimes extra state can be inferred, but it
|
|
may have to be stored between runs (possibly with SetLineState) or the code may have to
|
|
backtrack to a previous safe line - often something like a line that starts with a character
|
|
in the default style.
|
|
|
|
The SciTE.properties file is similar to properties files used for SciTE but are simpler.
|
|
The lexer to be run is defined with a lexer.{filepatterns} statement like:
|
|
lexer.*.d=d
|
|
|
|
Keywords may be defined with keywords settings like:
|
|
keywords.*.cxx;*.c=int char
|
|
keywords2.*.cxx=open
|
|
|
|
Substyles and substyle identifiers may be defined with settings like:
|
|
substyles.cpp.11=1
|
|
substylewords.11.1.*.cxx=map string vector
|
|
|
|
Other settings are treated as lexer or folder properties and forwarded to the lexer/folder:
|
|
lexer.cpp.track.preprocessor=1
|
|
fold=1
|
|
|
|
It is often necessary to set 'fold' in SciTE.properties to cause folding.
|
|
|
|
Properties can be set for a particular file with an "if $(=" or "match" expression like so:
|
|
if $(= $(FileNameExt);HeaderEOLFill_1.md)
|
|
lexer.markdown.header.eolfill=1
|
|
match Header*1.md
|
|
lexer.markdown.header.eolfill=1
|
|
|
|
More complex tests with additional configurations of keywords or properties can be performed
|
|
by creating another subdirectory with the different settings in a new SciTE.properties.
|
|
|
|
There is some support for running benchmarks on lexers and folders. The properties
|
|
testlexers.repeat.lex and testlexers.repeat.fold specify the number of times example
|
|
documents are lexed or folded. Set to a large number like testlexers.repeat.lex=10000
|
|
then run with a profiler.
|
|
|
|
A list of styles used in a lex can be displayed with testlexers.list.styles=1.
|