Wiki
Clone wikinmqual / NMqual_8
Overview of NMQual version 8.
Introduction
NMQual 8 has been completely redesigned to support NONMEM 7.2.0. Previous versions used Perl to emulate NONMEM's install and runtime scripts. This version uses Perl to provide a wrapper for those scripts.
What you need
- Perl 5.8 or later
- Perl package XML::XPath, installed
- NMQual version 8
- Anything you normally need to install NONMEM 'by hand' (like a fortran compiler!)
It's also useful to have a tool that lets you view XML files interactively, such as [http://www.google.com/chrome][Google Chrome].
What you get
NMQual 8.1.5 gives you ....
* autolog.pl
: discussed below
* nix/nm72.xml
: an example configuration file
* nix/setup72.diff
: a patch for setup72
* nix/nmfe72.diff
: a patch for nmfe72
* test/
: the usual qualification tests, reorganized
* doc/
: reference installation logs and qualification results
For Windows users ....
* win/nm72.xml
: an example configuration file for Windows
* win/SETUP72.bat
: a replacement for SETUP72.bat
* win/util/nmfe72original.bat
: a replacement for util/nmfe72original.bat
* win/infozip/zip.exe
: Info-Zip's zip utility
autolog.pl
The script autolog.pl
replaces nmqual.pl
. It is more general than its predecessor: so general, in fact, that it neither knows nor cares about NONMEM at all! autolog.pl
is just a logging processor of configurations that have a particular form (like that in nm72.xml
). It finds expressions in the configuration, evaluates them with a Perl system call, and logs the results. The first argument to autolog.pl
must be a path to a configuration file. All other command line arguments are available to the configured expressions.
the configuration file
The configuration file must be valid XML. It must have an element called config (hereafter, "the configuration"). The configuration can have child elements called alias, do, or to.
1. alias elements have an id attribute: this is a pattern that will be replaced by the element's text in all following aliases, expressions, and in attributes. Caution! The pattern is a Perl regular expression, so metacharacters like \ | ( ) [ { ^ $ *
+ ? . have powerful (possibly unwanted) effects.
2. do elements contain text that is passed to the system (i.e., a shell prompt) for evaluation. do elements have three optional attributes: on, in, and as.
* The on attribute specifies an operational mode. The expression will only be evaluated if there is no on attribute, or if the value of on is the first argument to autolog.pl
following the name of the configuration file.
* The in attribute specifies a directory in which to evaluate the expression. Perl will change to that directory, pass the expression to the system, capture the output, and then change back to the original directory.
* The as attribute tells autolog.pl
to expect results in a particular format. The default is plain text. if the value of as is "xml
" (case doesn't matter), then autolog.pl will treat the results as a stream of XML.
3. The to element specifies a file path to which to write the combined results. If the file does not exist, a new log with that name is created. If the file exists, and is valid XML, the log is appended as a child of the top element.
What actually happens
When you say perl autolog.pl config.xml [arguments]
at the command line, this is what autolog.pl
does.
1. config.xml
is read and parsed. If it is not valid XML, autolog.pl
dies. Otherwise, autolog.pl
grabs the first config element it can find, even if it is not the top element. Since autolog.pl
includes a copy of the configuration in its output, you can use the output as input!
2. By default, no mode is set, and output will go to standard output rather than a file.
3. autolog.pl
grabs the next command line argument (if it exists). If there is a match among the on attributes in config.xml
, this becomes the mode. If not, autolog.pl
puts the argument back: it will be argument number 1 later.
4. autolog.pl
throws out all elements with on attributes if there is no valid mode. If mode has been set, autolog.pl
keeps just the no-mode elements and those with matching mode (i.e., on value same as mode).
5. autolog.pl
processes each element.
* If any aliases have been registered, the appropriate substitutions are made to the element's text, and to any in attribute. The substitutions are evaluated in no particular order.
* If there are remaining command line arguments, they are substituted in text and in 'in' attributes. $_
is an alias for "all arguments" (space delimited). $_
1, $_
2 etc. are interpreted as the first argument, the second, and so on.
* If output is destined for a file, the expressions from do elements are printed to standard output, noting any specified directory.
* Expressions from do elements are passed to the system for evaluation, after changing to a specified directory, if any. The captured results are treated as plain text, or as XML if specified (as attribute).
* Expressions from alias elements are registered.
* Expressions from to elements become the default output path (file name).
6. The collection of results is formatted as XML and returned.
* Each do element, as substituted, is paired with a so element that contains the corresponding output (mnemonically, the results are "just so", or "system output"). These are wrapped in a job element.
* The do element retains its original in attribute, if any.
* job elements are listed in the order of evaluation, and wrapped in a logged element.
* The logged element has an on attribute that reports the mode, if any.
* The first child element of the logged element is a copy of the configuration used to generate the log.
* If a file was specified and exists, the logged element becomes the last top-level child of that file's first element.
* Otherwise, the logged element is wrapped in a generic log element and is printed to file (if specified) or standard output.
7. Best of all ... if any file operation fails, or if any expression evaluation fails, autolog.pl
dies and no log is created or appended. (Just what constitutes a failed evaluation is platform-dependent.)
an example
nix/nm72.xml
is an example of a config file that autolog.pl
can read. Essentially, it configures a set of related applications for installing and running NONMEM(r). Here we show blocks of code followed by related comments.
begin
<config>
alias
<alias id='folder'>/mnt/nm72</alias> <alias id='target'>/opt/NONMEM/nm72</alias> <alias id='origin'>/home/ubuntu/nmqual</alias>
folder
gives a path to the NONMEM source. We didn't call it source
because source
is used literally in the next alias.
* target
is where we want to build NONMEM.
* origin
is where nmqual.zip was unarchived. We didn't call it nmqual
because that's used literally elsewhere. We didn't call it .nmqual
because autolog.pl
treats this as a regular expression, and we're not sure what the dot would do.
do
<do>pwd</do> <do>date</do> <do>perl -e "use POSIX; print join qq(,),uname"</do> <do>echo $SHELL</do> <do>echo $USER</do> <do>echo $PATH</do>
autolog.pl
sees this configuration. So what should have that level of universality? Contextual metadata. We capture the directory context, the temporal context, the platform context, the interpreter context, the user context, and the path context.
install
<do on='install'>rm -rf origin/temp</do> <do on='install'>mkdir origin/temp</do> <do on='install'>cp -r folder/* origin/temp</do> <do on='install' in='origin'>patch -p0 < nix/setup72.diff</do> <do on='install'>rm -rf target</do> <do on='install'>mkdir target</do> <do on='install' in='origin/temp'>pwd</do> <do on='install' in='origin/temp'>ifort -V</do> <do on='install' in='origin/temp'>/bin/bash SETUP72 origin/temp target ifort y ar same rec q unzip nonmem72e.zip nonmem72r.zip</do> <do on='install'>rm -rf origin/temp</do> <do on='install'>mkdir target/nmqual</do> <do on='install'>cp origin/autolog.pl target/nmqual</do> <do on='install'>cp origin/nix/setup72.diff target/nmqual</do> <do on='install'>cp origin/nix/nmfe72.diff target/nmqual</do> <do on='install'>cp -r origin/test target/nmqual</do> <do on='install' in='target'>patch -p1 < nmqual/nmfe72.diff</do> <to on='install'>target/nmqual/log.xml</to>
autolog.pl
as a user with adequate permissions. We copy the NONMEM source to a temporary directory, and patch setup72. We delete the installation if it exists, re-create the directory, and capture some metadata. In the temporary directory, we run NONMEM's setup script exactly as instructed. Then we add an nmqual
directory and stick all our NMQual files there. We use the unix patch
utility to fix nmfe72. Thus changed, nmfe72 expects, as its first argument, "c", "e", or "ce": compile-only, execute-only, or compile-and-execute. You can skip this step if you don't need it. (R::metrumrg expects it).
Note that the block above has a to element, telling where the standard log should go.
run
<do on='run' in='$_2'>target/util/nmfe72 $_1 $_3.ctl $_3.lst</do> <do on='run' in='$_2' as='xml'>if [ -e $_3.xml ];then cat $_3.xml;else echo \<nofile/\>; fi</do> <do on='run' as='xml'>cat target/nmqual/log.xml</do> <to on='run'>$_2/$_3.log.xml</to>
one of c
, e
, or ce
, described earlier
a run directory
a run name
The first argument is passed directly to our modified nmfe72. The second argument tells where to run the expressions (directory context). The third argument is used to construct the file names. We know nmfe72 makes the *.xml
file, and we'd like to include it in our *log.xml
file, but we're not absolutely sure it's there, because this may have been a compile-only run. So in the second expression, we test for file existence, and make up a dummy xml element to dump if the file is not found. Note that the angle brackets are coded with character entity references, so they won't be mistaken for XML delimiters.
We attach the installation log to our run log, and write the run log to whatever the 5th argument was.
test
<do on='test' in='target/nmqual/test'>target/util/nmfe72 ce $_1.ctl $_1.lst</do> <do on='test' in='target/nmqual/test' as='xml'>cat $_1.xml</do>
runtest.pl
did. The single argument is the number of a test run. We run the application right in the test directory, where we know there are control streams with associated data. This time, we don't append the install log, because in fact we plan to append these results to the install log, and the install log does not need copies of itself. we do output NONMEM's xml file, marking it as XML, but we don't stuff it in a file.
qualify
<do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 3</do> <do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 4</do> <do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 5</do> <do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 6</do> <do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 7</do> <do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 8</do> <to on='qualify'>$_1</to>
test
mode, we can write a qualify
mode that uses it. NMQual's standard control tests 3 through 8 have been edited for use in this environment. We pass each as an argument to the test mode of log.xml
, run under autolog.pl (interpreted by Perl). If all the tests succeed, we write them to the specified file, possibly log.xml
itself (in which case log.xml
will be appended).
accept
<do on='accept'>echo $_</do> <to on='accept'>target/nmqual/log.xml</to>
accept
mode. Any arguments we pass, such as an acceptance statement, will be appended to log.xml
.
end
</config>
autolog.pl
only cares about alias, do, and to elements. But it always quotes its config entirely. Thus, you could insert a meta element to ensure that certain metadata is reproduced in the log.
An implementation
- Make sure you have perl installed, and XML::XPath, and a compiler. This example assumes Intel Fortran.
- Download nmqual-8.x.x.zip.
- Unarchive it using your system's tools.
- Fix the path aliases at the top of nix/nm72.xml or win/nm72.xml. Avoid spaces in file paths.
- At a command prompt, change to the nmqual directory.
- For Linux, Mac, Unix :
perl autolog.pl nix/nm72.xml install cd /opt/NONMEM/nm72/nmqual perl autolog.pl log.xml qualify qualify.xml perl autolog.pl log.xml accept This NONMEM instance meets my expectations.
Replace /opt/NONMEM/nm72
with the path of the NONMEM instance, if different.
Windows users: replace nix
with win
.
The qualify
and accept
steps are, of course, optional. Before administratively accepting an installation, you may wish to compare target/nmqual/qualify.xml
(yours) with origin/doc/nix/qualify.xml
(ours).
Updated