qstest.

sty
QuinScape Unit Test Package
version 1.7896
David Kastrup

2007/02/21
1 Using qstest
The basic idea of qstest is to let the user specify a number of tests that can be
performed either at package load time or while running a separate test le through
L
A
T
E
X. If you are writing .dtx les, it is a good idea to use Docstrip modules
for specifying which lines are to be used for testing. The le qstest.dtx from
which both the style le as well as this documentation have been generated has
been done in this manner.
Since the following tests should be ignored when the dtx le is compiled itself,
we use this for skipping over the tests:
1 dtx
2 \iffalse
3 /dtx
A standalone test le actually does not need a preamble. We can load the
packages with \RequirePackage and just go ahead. Let us demonstrate how to
build such a test le by testing the qstest package itself:
4 test
5 \RequirePackage{qstest}
1.1 Pattern and keyword lists
Check section Match patterns and targets of the makematch package for an
explanation of the comma-separated pattern and keyword lists. In a nutshell, both
are lists of arbitrary material that is not expanded but rather used in sanitized
(printable) form. Patterns may contain wildcard characters * matching zero or
more characters, and may be preceded by ! in order to negate a match from an
earlier pattern in the pattern list. Leading spaces before an item in either list are
discarded.

[email protected], QuinScape GmbH

1
1.2 Specifying test sets
This macro species a pattern list matched to tests keyword lists in order to \IncludeTests
determine the tests to be included in this test run. The characters * and ! are
interpreted as explained above.
For example,
\IncludeTests{*, !\cs}
will run all tests except those that have a test keyword of \cs in their list of
keywords. It is a good convention to specify the principal macro or environment
to be tested as the rst keyword.
The default is to include all tests. If you are interspersing possibly expensive
tests with your source le, you might want to specify something like
\IncludeTests{*, !expensive}
or even
\IncludeTests{}
in your document preamble, and then possibly override this on the command line
with
latex "\AtBeginDocument{\IncludeTests{*}}\input{file}"
or similar for getting a more complete test.
This denes test patterns that will throw an error when failing. A test that \TestErrors
throws an error will not also add a warning to the standard log le with extension
log since that would be redundant.
The default is \TestErrors{*, !fails}, have all tests that are not marked
as broken throw an error when they fail.
The throwing of errors does not depend on the logging settings (see below) for
the default log le.
This macro receives three arguments. The rst is the lename extension of a \LogTests
log le (the extension log is treated specially and uses package warning and info
commands to log test failures and passes, respectively). The second is a keyword
list that indicates which passed tests are to be logged. The third is a keyword list
specifying which failed tests are to be logged. Let us open a le logging everything:
6 \LogTests{lgout}{*}{*}
The choice of lgout is made to make it possible to also have lgin for comparison
purposes: the latter would be an lgout le from a previous, denitive run,
renamed and checked into version control, for the sake of being able to compare
the log output from dierent versions.
An already open log le stays open and just changes what is logged. By default,
the standard log (pseudo-)le is already open and logs everything.
Passed and failed tests are not completely symmetric with regard to logging:
test failures are logged and/or indicated on the individual failed assertions, while
a successful test is only logged and/or indicated in summary.
2
You can explicitly close a log le if you want to reread it in the course of \LogClose
processing, or call an executable that would process it. The standard le with
extension log will not actually get closed and ushed if you do this (though
logging would stop on it), but all others might. The actual example for this
follows after the tests. You can reopen a closed log le using \LogTests. It will
then get rewritten from the beginning (with the exception for the standard log
le, of course).
1.3 The tests
Tests are performed within the qstest environment. The environment gets two qstest
arguments. The rst is the name of the test recorded in the log le. The second
is a list of test keywords used for deciding which tests are performed and logged.
Before delving in the details of what kind of tests you can perform in this en-
vironment, we list the various commands that are given patterns and thus control
what kind of tests are performed and logged.
This is the workhorse for checking that values, dimensions, macros and other \Expect
things are just like the test designer would expect them to be.
This macro basically receives two brace-delimited arguments
1
and checks that
they are equal after being passed through \def and sanitized. This means that you
cant normally use # except when followed by a digit (all from 1 to 9 are allowed)
or #. If you precede one of those arguments with * it gets passed through through
\edef instead of \def. There may also be additional tokens like \expandafter
before the opening brace. Note that the combination of \edef and \thetoken
variable can be used to pass through # characters without interpretation. eT
E
X
provides a similar eect with \unexpanded. So if you want to compare a token
list that may contain isolated hash characters, you can do so by writing something
like
7 etex
8 \begin{qstest}{# in isolation}{\Expect, #, \unexpanded}
9 \toks0{# and #}
10 \Expect*{\the\toks0}*{\unexpanded{# and #}}
11 \end{qstest}
12 /etex
Since the sanitized version will convert # macro parameters to the string ##, you
might also do this explicitly (and without eT
E
X) as
13 \begin{qstest}{# in isolation 2}{\Expect, #, \string}
14 \toks0{# and #}
15 \Expect*{\the\toks0}*{\string#\string# and \string#\string#}
16 \end{qstest}
If the token register is guaranteed to only contain proper # characters that are
either followed by another # or a digit, you can let the normal interpretation of a
macro parameter for \def kick in and use this as
1
The arguments are collected with a token register assignment. This gives several options
for specifying them, including giving a token register without braces around it. It also makes it
possible to precede the balanced text with \expandafter and similar expandable stu.
3
17 \begin{qstest}{# as macro parameter}{\Expect, #}
18 \toks0{\def\xxx#1{}}
19 \Expect\expandafter{\the\toks0}{\def\xxx#1{}}
20 \end{qstest}
In this manner, #1 is interpreted (and sanitized) as a macro parameter on both
sides, and consequently no doubling of # occurs.
Before the comparison is done, both arguments are sanitized, converted into
printing characters with standardized catcodes throughout
2
. A word of warning:
both sanitization as well as using \meaning still depend on catcode settings, since
single-letter control sequences (made from a catcode 11 letter) are followed by a
space, and other single-character control sequences are not. For that reason, a
standalone test le for L
A
T
E
X class or package les will usually need to declare
21 \makeatletter
in order to make @ a letter, like it usually is in such les.
All of the of the following expectations would turn out correct:
22 \begin{qstest}{Some LaTeX definitions}{\Expect}
23 \Expect*{\meaning\@gobble}{\long macro:#1->}
24 \Expect*{\the\maxdimen}{16383.99998pt}
25 \end{qstest}
Note that there is no way to convert the contents of a box into a printable rendition,
so with regard to boxes, you will be mostly reduced to check that dimensions of a
box meet expectations.
1.4 Expecting ifthen conditions
This is used for evaluating a condition as provided by the ifthen package. See \ExpectIfThen
its docs for the kind of condition that is possible there. You just specify one
argument: the condition that you expect to be true. Here is an example:
26 \RequirePackage{ifthen}
27 \begin{qstest}{\ExpectIfThen}{\ExpectIfThen}
28 \ExpectIfThen{\lengthtest{\maxdimen=16383.99998pt}\AND
29 \maxdimen>1000000000}
30 \end{qstest}
1.5 Dimension ranges
Sometimes we want to check whether some dimension is not exactly like some \InRange
value, but rather in some range. We do this by specifying as the second argument
to \Expect an articial macro with two arguments specifying the range in question.
This will make \Expect succeed if its rst argument is in the range specied by
the two arguments to \InRange.
The range is specied as two T
E
X dimens. If you use a dimen register and
you want to have a possible error message display the value instead of the dimen
register, you can do so by using the * modier before \InRange (which will cause
2
Spaces get catcode 10, all other characters catcode 12.
4
the value to be expanded before printing and comparing) and put \the before the
dimen register since such register are not expandible by themselves.
Here are some examples:
31 \begin{qstest}{\InRange success}{\InRange}
32 \dimen@=10pt
33 \Expect*{\the\dimen@}\InRange{5pt}{15pt}
34 \Expect*{\the\dimen@}\InRange{10pt}{15pt}
35 \Expect*{\the\dimen@}\InRange{5pt}{10pt}
36 \end{qstest}
37 \begin{qstest}{\InRange failure}{\InRange, fails}
38 \dimen@=10pt \dimen@ii=9.99998pt
39 \Expect*{\the\dimen@}\InRange{5pt}{\dimen@ii}
40 \dimen@ii=10.00002pt
41 \Expect*{\the\dimen@}*\InRange{\the\dimen@ii}{15pt}
42 \end{qstest}
This requires eT
E
Xs arithmetic and will not be available for versions built without \NearTo
eT
E
X support. The macro is used in lieu of an expected value and is similar to
\nRange in that it is a pseudovalue to be used for the second argument of \Expect.
It makes \Expect succeed if its own mandatory argument is close to the rst
argument from \Expect, where closeness is dened as being within 0.05pt. This
size can be varied by specifying a dierent one as optional argument to \NearTo.
Here is a test:
43 etex
44 \begin{qstest}{\NearTo success}{\NearTo}
45 \dimen@=10pt
46 \Expect*{\the\dimen@}\NearTo{10.05pt}
47 \Expect*{\the\dimen@}\NearTo{9.95pt}
48 \Expect*{\the\dimen@}\NearTo[2pt]{12pt}
49 \Expect*{\the\dimen@}\NearTo[0.1pt]{9.9pt}
50 \end{qstest}
51 \begin{qstest}{\NearTo failure}{\NearTo, fails}
52 \dimen@=10pt
53 \Expect*{\the\dimen@}\NearTo{10.05002pt}
54 \Expect*{\the\dimen@}\NearTo[1pt]{11.00001pt}
55 \end{qstest}
56 /etex
1.6 Saved results
The purpose of saved results is to be able to check that the value has remained
the same over passes. Results are given a unique label name and are written to an
auxiliary le where they can be read in for the sake of comparison. One can use
the normal aux le for this purpose, but it might be preferable to use a separate
dedicated le. That way it is possible to input a versioned copy of this le and
have a xed point of reference rather than the last run.
While the aux le is read in automatically at the beginning of the document,
this does not happen with explicitly named les. You have to read them in yourself,
5
preferably using
\InputIfFileExists{lename}{}{}
so that no error is thrown when the le does not yet exist.
This gets one argument specifying which le name to use for saving results. \SaveValueFile
If this is specied, a special le is opened. If \SaveValueFile is not called,
the standard aux le is used instead, but then you can only save values after
\begin{document}. \jobname.qsout seems like a useful le name to use here
(the extension out is already in use by PDFT
E
X).
57 \begin{qstest}{\SavedValue}{\SavedValue}
58 \SaveValueFile{\jobname.qsout}
If this were a real test instead of just documentation, we probably would have
written something like
\InputIfFileExists{\jobname.qsin}{}{}
rst in order to read in values from a previous run. The given le would have been
a copy of a previous qsout le, possibly checked into version control in order to
make sure behavior is consistent across runs. If it is an error to not have such a
le (once you have established appropriate testing), you can just write
\input{\jobname.qsin}
instead, of course.
This macro takes no argument and will close a value save le if one is open \CloseValueFile
(using this has no eect if no le has been opened and values are saved on the
aux le instead). Well demonstrate use of it later. It is probably only necessary
for testing qstest itself (namely, when you read in saved values in the same run),
or when you do the processing/comparison with a previous version by executing
commands via T
E
Xs \write18 mechanism.
This gets the label name as rst argument. If you are using the non-eT
E
X- \SaveValue
version, the label name gets sanitized using \string and so cant deal with non-
character material except at its immediate beginning. The eT
E
X-version has no
such constraint.
The second argument is the same kind of argument as \Expect expects, namely
something suitable for a token register assignment which is passed through \def
if not preceded by *, and by \edef if preceded by *. The value is written out to
the save le where it can be read in afterwards.
Let us save a few values under dierent names now:
59 \SaveValue{\InternalSetValue}*{\meaning\InternalSetValue}
60 \SaveValue{\IncludeTests}*{\meaning\IncludeTests}
61 \SaveValue{whatever}*{3.1415}
62 \SaveValue{\maxdimen}*{\the\maxdimen}
A call to this macro is placed into the save le for each call of \SaveValue. The \InternalSetValue
details are not really relevant: in case you run into problems while inputting the
save le, it might be nice to know.
This is used for retrieving a saved value. When used as the second argument \SavedValue
6
to \Expect, it will default to the value of the rst argument to \Expect unless it
has been read in from a save le. This behavior is intended for making it easy to
add tests and saved values and not get errors at rst, until actually values from a
previous test become available.
Consequently, the following tests will all turn out true before we read in a
le, simply because all the saved values are not yet dened and default to the
expectations:
63 \Expect{Whatever}\SavedValue{\InternalSetValue}
64 \Expect[\IncludeTests]{Whatever else}\SavedValue{\IncludeTests}
65 \Expect[whatever]{2.71828}\SavedValue{whatever}
66 \Expect[undefined]{1.618034}\SavedValue{undefined}
After closing and rereading the le, well need to be more careful with our expecta-
tions, but the last line still works since there still is no saved value for undened.
67 \CloseValueFile
68 \input{\jobname.qsout}
69 \Expect*{\meaning\InternalSetValue}\SavedValue{\InternalSetValue}
70 \Expect[\IncludeTests]*{\meaning\IncludeTests}%
71 \SavedValue{\IncludeTests}
72 \Expect[whatever]{3.1415}\SavedValue{whatever}
73 \Expect[undefined]{1.618034}\SavedValue{undefined}
74 \end{qstest}
Ok, and now lets take the previous tests which succeeded again and let them fail
now that the variables are dened:
75 \begin{qstest}{\SavedValue failure}{\SavedValue,fails}
76 \Expect{Whatever}\SavedValue{\InternalSetValue}
77 \Expect[\IncludeTests]{Whatever else}\SavedValue{\IncludeTests}
78 \Expect{2.71828}\SavedValue{whatever}
79 \end{qstest}
1.7 Checking for call sequences
The environment ExpectCallSequence tells the test system that macros are going ExpectCallSequence
to be called in a certain order and with particular types of arguments.
It gets as an argument the expected call sequence. The call sequence contains
entries that look like a macro denition: starting with the macro name followed
with a macro argument list and a brace-enclosed substitution text that gets exe-
cuted in place of the macro. The argument list given to this macro substitution
will get as its rst argument a macro with the original denition of the control
sequence, so you can get at the original arguments for this particular macro call
starting with #2. As a consequence, if you specify no arguments at all and an
empty replacement text for the substitution, the original macro gets executed
with the original arguments.
If you want to get back from the control sequence with the original meaning in \CalledName
#1 to the original macro name, you can use \CalledName on it. This will expand
to the original control sequence name, all in printable characters t for output or
7
typesetting in a typewriter font (or use in \csname), but without leading backslash
character. You can get to the control sequence itself by using
\csname \CalledName#1\endcsname
and to a printable version including backslash by using
\expandafter \string \csname \CalledName#1\endcsname
Going into more detail, a substitution function is basically dened using
\protected \long \def
so it will not usually get expanded except when hit with \expandafter or actually
being executed. Note that you cant use this on stu that has to work at expansion
time. This really works mainly with macros that would also be suitable candidates
for \DeclareRobustCommand.
It is also a bad idea to trace a conditional in this manner: while the substitution
could be made to work when being executed, it will appear like an ordinary macro
when being skipped, disturbing the conditional nesting.
Only macros occuring somewhere in the call sequence will get tracked, other
macros are not aected. The environment can actually get nested, in which case
the outer sequences will get tracked independently from the inner sequence.
This makes it possible to use ExpectCallSequence in order to spoof, for ex-
ample, both input consuming and output producing macros without knowing the
exact relationship of both.
Apart from specifying macro calls, the call sequence specication can contain
special characters that also carry meaning:
If this is set in the call sequence, it places the initial parsing state here. This will
make it an error if non-matching entries occur at the start of the sequence,
which otherwise is eectively enclosed with
.{}*(sequence).{}*
meaning that nonmatching entries before the rst and after the last matching
item of the sequence are ignored by default (this makes it closer to normal
regexp matchers). Since the matching will then start at , you can put
macros before that position that you want to be agged if they occur in
the sequence, even when they are mentioned nowhere else (macros which
would be an error if actually called). Also available as the more customary
^ character, but that tends to behave worse in L
A
T
E
X-aware editors.
This indicates the last call sequence element to be matched. If any traced
macros appear after this point, an error will get generated. Any immediately
following call sequence entries will not get reached.
. A single dot indicates a wildcard: any of the tracked control sequences might
occur here. You still have to follow this with macro arguments and a braced
replacement text. Wildcards are considered as a fallback when nothing else
matches.
8
(. . . ) Parens may be used for grouping alternatives and/or combining items for
the sake of repeating specications, of which there are three:
? If a question mark follows either a macro call, wildcard call, parenthesized group,
or call sequence end, the item before it is optional.
+ A plus sign following an item means that this item may be repeated one or more
times.
* An asterisk following an item means that this item may be repeated zero or
more times.
| A vertical bar separates alternatives. Alternatives extend as far as possible, to
the next bar, to an enclosing paren group, or to the start and/or end of the
whole call sequence specication if nothing else intervenes.
Note that opposed to traditional regexp evaluation, no backtracking is employed:
at each point in the call sequence, the next match is immediately chosen and a
choice can (for obvious reasons) not be reverted. It is the task of the user to
specify a call sequence in a suciently non-ambiguous manner that will make the
call sequence tracing not pick dead ends.
80 \begin{qstest}{ExpectCallSequence}{ExpectCallSequence}
81 \def\e{e} \def\f{f}
82 \def\g{g} \def\h{h}
83 \begin{ExpectCallSequence}{\e#1{%
84 \Expect\expandafter{\csname\CalledName#1\endcsname}{\e }%
85 \Expect*{\meaning#1}{macro:->e}}+\f#1{}}
86 \e \e \e \e \f
87 \end{ExpectCallSequence}
88 \end{qstest}
1.8 Ending a standalone test le
One nishes a standalone test le by calling the L
A
T
E
X control sequence \quit.
This stops processing even when we did not actually get into a document. We dont
actually do this here since there will be further tests in the full documentation
le. However, we will now close the log le we opened for this demonstration.
89 \LogClose{lgout}
90 /test
And now we will show the resulting standalone log le:
91 Passed: ## in isolation
92 Passed: ## in isolation 2
93 Passed: ## as macro parameter
94 Passed: Some LaTeX definitions
95 Passed: \ExpectIfThen
96 Passed: \InRange success
97 Failed: \InRange failure
98 \Expect: \the \dimen@
9
99 <in [5pt..\dimen@ii ]
100 >10.0pt
101 Failed: \InRange failure
102 \Expect: \the \dimen@
103 <in [10.00002pt..15pt]
104 >10.0pt
105 Passed: \NearTo success
106 Failed: \NearTo failure
107 \Expect: \the \dimen@
108 <near[0.05pt] to 10.05002pt
109 >10.0pt
110 Failed: \NearTo failure
111 \Expect: \the \dimen@
112 <near[1pt] to 11.00001pt
113 >10.0pt
114 Passed: \SavedValue
115 Failed: \SavedValue failure
116 \Expect: Whatever
117 <macro:->\begingroup \let \do \@makeother \dospecials \endlinechar \
118 \catcode \^7 \qst@setsavedii
119 >Whatever
120 Failed: \SavedValue failure
121 \Expect: \IncludeTests
122 <macro:->\MakeMatcher [,]\qst@includecheck
123 >Whatever else
124 Failed: \SavedValue failure
125 \Expect: 2.71828
126 <3.1415
127 >2.71828
128 Passed: ExpectCallSequence
We can now stop skipping if we are compiling the dtx le standalone.
129 dtx
130 \fi
131 /dtx
10