1. metrumrg Avatar metrumrg
  2. project
  3. nmqual

Wiki

Clone wiki

nmqual / NMqual_8

View History

Overview of NMQual version 8.

Introduction

NMQual 8 has been completely redesigned to support NONMEM 7.2.0. Previous versions used Perl to emulate NONMEM's install and runtime scripts. This version uses Perl to provide a wrapper for those scripts.

What you need

  • Perl 5.8 or later
  • Perl package XML::XPath, installed
  • NMQual version 8
  • Anything you normally need to install NONMEM 'by hand' (like a fortran compiler!)

It's also useful to have a tool that lets you view XML files interactively, such as [http://www.google.com/chrome][Google Chrome].

What you get

NMQual 8.1.5 gives you .... * autolog.pl: discussed below * nix/nm72.xml: an example configuration file * nix/setup72.diff: a patch for setup72 * nix/nmfe72.diff: a patch for nmfe72 * test/: the usual qualification tests, reorganized * doc/: reference installation logs and qualification results

For Windows users .... * win/nm72.xml: an example configuration file for Windows * win/SETUP72.bat: a replacement for SETUP72.bat * win/util/nmfe72original.bat: a replacement for util/nmfe72original.bat * win/infozip/zip.exe: Info-Zip's zip utility

autolog.pl

The script autolog.pl replaces nmqual.pl. It is more general than its predecessor: so general, in fact, that it neither knows nor cares about NONMEM at all! autolog.pl is just a logging processor of configurations that have a particular form (like that in nm72.xml). It finds expressions in the configuration, evaluates them with a Perl system call, and logs the results. The first argument to autolog.pl must be a path to a configuration file. All other command line arguments are available to the configured expressions.

the configuration file

The configuration file must be valid XML. It must have an element called config (hereafter, "the configuration"). The configuration can have child elements called alias, do, or to. 1. alias elements have an id attribute: this is a pattern that will be replaced by the element's text in all following aliases, expressions, and in attributes. Caution! The pattern is a Perl regular expression, so metacharacters like \ | ( ) [ { ^ $ * + ? . have powerful (possibly unwanted) effects. 2. do elements contain text that is passed to the system (i.e., a shell prompt) for evaluation. do elements have three optional attributes: on, in, and as. * The on attribute specifies an operational mode. The expression will only be evaluated if there is no on attribute, or if the value of on is the first argument to autolog.pl following the name of the configuration file. * The in attribute specifies a directory in which to evaluate the expression. Perl will change to that directory, pass the expression to the system, capture the output, and then change back to the original directory. * The as attribute tells autolog.pl to expect results in a particular format. The default is plain text. if the value of as is "xml" (case doesn't matter), then autolog.pl will treat the results as a stream of XML. 3. The to element specifies a file path to which to write the combined results. If the file does not exist, a new log with that name is created. If the file exists, and is valid XML, the log is appended as a child of the top element.

What actually happens

When you say perl autolog.pl config.xml [arguments] at the command line, this is what autolog.pl does. 1. config.xml is read and parsed. If it is not valid XML, autolog.pl dies. Otherwise, autolog.pl grabs the first config element it can find, even if it is not the top element. Since autolog.pl includes a copy of the configuration in its output, you can use the output as input! 2. By default, no mode is set, and output will go to standard output rather than a file. 3. autolog.pl grabs the next command line argument (if it exists). If there is a match among the on attributes in config.xml, this becomes the mode. If not, autolog.pl puts the argument back: it will be argument number 1 later. 4. autolog.pl throws out all elements with on attributes if there is no valid mode. If mode has been set, autolog.pl keeps just the no-mode elements and those with matching mode (i.e., on value same as mode). 5. autolog.pl processes each element. * If any aliases have been registered, the appropriate substitutions are made to the element's text, and to any in attribute. The substitutions are evaluated in no particular order. * If there are remaining command line arguments, they are substituted in text and in 'in' attributes. $_ is an alias for "all arguments" (space delimited). $_1, $_2 etc. are interpreted as the first argument, the second, and so on. * If output is destined for a file, the expressions from do elements are printed to standard output, noting any specified directory. * Expressions from do elements are passed to the system for evaluation, after changing to a specified directory, if any. The captured results are treated as plain text, or as XML if specified (as attribute). * Expressions from alias elements are registered. * Expressions from to elements become the default output path (file name). 6. The collection of results is formatted as XML and returned. * Each do element, as substituted, is paired with a so element that contains the corresponding output (mnemonically, the results are "just so", or "system output"). These are wrapped in a job element. * The do element retains its original in attribute, if any. * job elements are listed in the order of evaluation, and wrapped in a logged element. * The logged element has an on attribute that reports the mode, if any. * The first child element of the logged element is a copy of the configuration used to generate the log. * If a file was specified and exists, the logged element becomes the last top-level child of that file's first element. * Otherwise, the logged element is wrapped in a generic log element and is printed to file (if specified) or standard output. 7. Best of all ... if any file operation fails, or if any expression evaluation fails, autolog.pl dies and no log is created or appended. (Just what constitutes a failed evaluation is platform-dependent.)

an example

nix/nm72.xml is an example of a config file that autolog.pl can read. Essentially, it configures a set of related applications for installing and running NONMEM(r). Here we show blocks of code followed by related comments.

begin

<config>
All other instructions must be nested within a config element. We can do namespaces and all that stuff right here, if it becomes necessary.

alias

<alias id='folder'>/mnt/nm72</alias>
<alias id='target'>/opt/NONMEM/nm72</alias>
<alias id='origin'>/home/ubuntu/nmqual</alias>
Aliases give you a powerful way to simplify the configuration. They accumulate, so the text of one alias can invoke the id of a former. But don't get too fancy: even though they accumulate in document order, they are currently applied in a random order (easier to implement in Perl). * folder gives a path to the NONMEM source. We didn't call it source because source is used literally in the next alias. * target is where we want to build NONMEM. * origin is where nmqual.zip was unarchived. We didn't call it nmqual because that's used literally elsewhere. We didn't call it .nmqual because autolog.pl treats this as a regular expression, and we're not sure what the dot would do.

do

<do>pwd</do>
<do>date</do>
<do>perl -e "use POSIX; print join qq(,),uname"</do>
<do>echo $SHELL</do>
<do>echo $USER</do>
<do>echo $PATH</do>
This is just a bunch of things to do. Since there is no on attribute, these will be done every time autolog.pl sees this configuration. So what should have that level of universality? Contextual metadata. We capture the directory context, the temporal context, the platform context, the interpreter context, the user context, and the path context.

install

<do on='install'>rm -rf origin/temp</do>
<do on='install'>mkdir origin/temp</do>
<do on='install'>cp -r folder/* origin/temp</do>
<do on='install' in='origin'>patch -p0 &lt; nix/setup72.diff</do>
<do on='install'>rm -rf target</do>
<do on='install'>mkdir target</do>
<do on='install' in='origin/temp'>pwd</do>
<do on='install' in='origin/temp'>ifort -V</do>
<do on='install' in='origin/temp'>/bin/bash SETUP72 origin/temp target ifort y ar same rec q unzip nonmem72e.zip nonmem72r.zip</do>
<do on='install'>rm -rf origin/temp</do>
<do on='install'>mkdir target/nmqual</do>
<do on='install'>cp origin/autolog.pl target/nmqual</do>
<do on='install'>cp origin/nix/setup72.diff target/nmqual</do> 
<do on='install'>cp origin/nix/nmfe72.diff target/nmqual</do> 
<do on='install'>cp -r origin/test target/nmqual</do> 
<do on='install' in='target'>patch -p1 &lt; nmqual/nmfe72.diff</do> 
<to on='install'>target/nmqual/log.xml</to>
The block above installs NONMEM(r). It's up to you to invoke autolog.pl as a user with adequate permissions. We copy the NONMEM source to a temporary directory, and patch setup72. We delete the installation if it exists, re-create the directory, and capture some metadata. In the temporary directory, we run NONMEM's setup script exactly as instructed. Then we add an nmqual directory and stick all our NMQual files there. We use the unix patch utility to fix nmfe72. Thus changed, nmfe72 expects, as its first argument, "c", "e", or "ce": compile-only, execute-only, or compile-and-execute. You can skip this step if you don't need it. (R::metrumrg expects it).

Note that the block above has a to element, telling where the standard log should go.

run

<do on='run' in='$_2'>target/util/nmfe72 $_1 $_3.ctl $_3.lst</do>
<do on='run' in='$_2' as='xml'>if [ -e $_3.xml ];then cat $_3.xml;else echo \&lt;nofile/\&gt;; fi</do>
<do on='run' as='xml'>cat target/nmqual/log.xml</do>
<to on='run'>$_2/$_3.log.xml</to>
Above, we write a small application to run the configured NONMEM installation. The application expects three arguments.

one of c, e, or ce, described earlier

a run directory

a run name

The first argument is passed directly to our modified nmfe72. The second argument tells where to run the expressions (directory context). The third argument is used to construct the file names. We know nmfe72 makes the *.xml file, and we'd like to include it in our *log.xml file, but we're not absolutely sure it's there, because this may have been a compile-only run. So in the second expression, we test for file existence, and make up a dummy xml element to dump if the file is not found. Note that the angle brackets are coded with character entity references, so they won't be mistaken for XML delimiters.

We attach the installation log to our run log, and write the run log to whatever the 5th argument was.

test

<do on='test' in='target/nmqual/test'>target/util/nmfe72 ce $_1.ctl $_1.lst</do>
<do on='test' in='target/nmqual/test' as='xml'>cat $_1.xml</do>
These elements create another small application that looks much like the run-mode application. It also works much like runtest.pl did. The single argument is the number of a test run. We run the application right in the test directory, where we know there are control streams with associated data. This time, we don't append the install log, because in fact we plan to append these results to the install log, and the install log does not need copies of itself. we do output NONMEM's xml file, marking it as XML, but we don't stuff it in a file.

qualify

<do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 3</do>
<do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 4</do>
<do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 5</do>
<do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 6</do>
<do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 7</do>
<do on='qualify' in='target/nmqual' as='xml'>perl autolog.pl log.xml test 8</do>
<to on='qualify'>$_1</to>
Now that we have a test mode, we can write a qualify mode that uses it. NMQual's standard control tests 3 through 8 have been edited for use in this environment. We pass each as an argument to the test mode of log.xml, run under autolog.pl (interpreted by Perl). If all the tests succeed, we write them to the specified file, possibly log.xml itself (in which case log.xml will be appended).

accept

<do on='accept'>echo $_</do>
<to on='accept'>target/nmqual/log.xml</to>
For administrative purposes, we write ourselves an accept mode. Any arguments we pass, such as an acceptance statement, will be appended to log.xml.

end

</config>
The close tag ends our config file. Note: you could have all manner of other elements in this file that serve other purposes. autolog.pl only cares about alias, do, and to elements. But it always quotes its config entirely. Thus, you could insert a meta element to ensure that certain metadata is reproduced in the log.

An implementation

  1. Make sure you have perl installed, and XML::XPath, and a compiler. This example assumes Intel Fortran.
  2. Download nmqual-8.x.x.zip.
  3. Unarchive it using your system's tools.
  4. Fix the path aliases at the top of nix/nm72.xml or win/nm72.xml. Avoid spaces in file paths.
  5. At a command prompt, change to the nmqual directory.
  6. For Linux, Mac, Unix : 
    perl autolog.pl nix/nm72.xml install
cd /opt/NONMEM/nm72/nmqual
perl autolog.pl log.xml qualify qualify.xml
perl autolog.pl log.xml accept This NONMEM instance meets my expectations.

Replace /opt/NONMEM/nm72 with the path of the NONMEM instance, if different.

Windows users: replace nix with win.

The qualify and accept steps are, of course, optional. Before administratively accepting an installation, you may wish to compare target/nmqual/qualify.xml (yours) with origin/doc/nix/qualify.xml (ours).

Updated