youys
/
ant

<?xml version="1.0"?>
<document>

  <properties>
    <author email="conor@apache.org">Conor MacNeill</author>
    <title>Mutant Design Notes</title>
  </properties>

<body>

<section name="Mutant Design Notes">
<p>
This is a brief, albeit rambling description of Mutant.
Mutant has many experimental ideas which may or may not prove useful.
I'll try to describe what is there and let anyone who is interested
comment. Mutant is still immature. You'll notice that there is, at this
time, just one task, a hacked version of the echo task, which I have
been using to test out ideas. Most tasks would end up being pretty
similar to their Ant 1.x version.
</p>

<p>
OK, let me start with some of the motivating requirements. There are of
coure many Ant2 requirements but I want to focus on these two for now.
Mutant does also address many of the other Ant2 requirements. 
</p>

<p>
I'll use the terms Ant and mutant somewhat interchangeably - just
habit, not an assumption of any sort.
</p>

<p>
One of the things which is pretty difficult in Ant 1.x is the
management of classpaths and classloaders. For example, today the
antlr task requires the antlr classes in the classpath used to start
ant. I'm talking here about the classpath built up in the ant.bat/ant
script launchers. At the same time, the checkstyle task
which uses antlr won't run if the antlr classes are in the classpath
because then those classes cannot &quot;see&quot; the classes in the taskdef's
classpath.
</p>

<p>
Another requirement I have is extensibility. In Ant 1.x this is
difficult because whenever a new type is created, each task which
needs to support this type must be changed to provide the new addXXX
method. The ejbjar task is on example of this problem with its concept of vendor
specific tools. The zip/jar task, with its support for different types
of fileset, is another. The addition of the classfileset to Ant requires
a change to the zip task.
</p>
</section>

<section name="Mutant Initialization">
<p>
Mutant defines a classloader hierarchy somewhat similar to that used
in Tomcat 4. Tasks join into this hierarchy at a particular point to
ensure they have visibility of the necessary interface classes and no
visibility of the Ant core itself. There is nothing particularly novel
about this approach, but tasks are able to request certain additional resources
as we will see later.
</p>

<p>
Mutant starts with two jars. One is the start.jar which contains just
one class, Main.java which establishes the initial configuration and
then runs the appropriate front end command line class. If a different
front end was desired, a different launch class, in its own jar, would
be used. This would perhaps configure the classloader hierarchy somewhat
differently and start the approriate GUI front end class.
</p>

<p> 
The second jar, init.jar, provides a number of initialisation utilities. These
are used by Main.java to setup Ant and would also be used by any other front end
to configure Ant. The important class here is the
InitConfig which communicates the state of Ant at startup into the the core of
Ant when it starts up. Main determines the location of ANT_HOME based on the
location of the start classes and then populates the InitConfig with both
classloaders and information about the location of various jars and config
files.
</p>

<p> 
At the top of the classloader hierarchy 
are the bootstrap and system classloaders. I won't really
distinguish between these in mutant. Combined they provide the JDK
classes, plus the classes from the init and start jars. One objective is
to keep the footprint of the init and start jars small so they do not
require any external classes, which may then become visible lower in the
hierarchy. Main does not explicitly create these loaders, of course, but
just adds a reference to the init config as system class loader 
</p>

<p> 
The next jar is for the common area. This provides interface definitions
and utility classes for use by both the core and by tasks/types etc. It
is loaded from ANT_HOME/lib/common/*.jar. Typically this is just
lib/common/common.jar but any other jars in here are loaded. This
pattern is used in the construction of all of the classloaders.
</p>

<p> 
Next up is the core loader. It includes the lib/antcore/antcore.jar plus
any others including the XML parser jars. Mutant's core does not assume that
the project model will come from an XML description but XML facilities
are needed in the core for reading in Ant library defs and config files.
The parser jar locations are also stored in the init config. This lets
the jars be added to any Ant library that wants to use Ant's XML parser
rather than providing its own. Similarly tools.jar's location is
determined automatically and added to the config for use by tasks which
request it. I'll go into more detail when discussing the antlib processing. 
</p>

<p> 
The final jar that is loaded is the jar for the frontend - cli.jar. This
is not passed in init config since these classes are not visible to the
core and are not needed by it. So the hierarchy is
<pre> 
jdk classes
    |
start/init
    |
 common
    |
 antcore
    |
   cli
</pre>
</p>

<p> 
Task classloaders generally will come in at common, hiding the core classes, front
end and XML parser classes from tasks.
</p>

<p> 
Once Main has setup the initConfig, it creates the front end commandline
class and launches mutant proper, passing it the command line args and
the init config. 
</p>

<p> 
A GUI would typically replace start.jar and the cli.jar with its own
versions which manage model construction from GUI processes rather than
from XML files. It may be possible to move some of Main.java's
processing into init.jar if it is useful to other front ends. I haven't
looked at that balance.
</p>
</section>

<section name="Mutant Frontend">
<p>
The front end is responsible for coordinating execution of Ant. It
manages command line arguments, builds a model of the Project to be
evaluated and coordinates the execution services of the core. cli.jar
contains not only the front-end code but also the XML parsing code for
building a project model from an XML description. Other front ends may
choose to build project models in different ways. Commandline is pretty
similar to Ant 1.x's Main.java - it handles arguments, building loggers,
listeners, defines, etc - actually I haven't fully implemented 
command line defines in
mutant yet but it would be similar to Ant 1.x. 
</p>

<p>
Commandline then moves to building a project model from the XML
representation. I have just expanded the approach in Ant 1's
ProjectHelper for XML parsing, moving away from a stack of inner classes.
The classes in the front end XML parsing use some XML utility base
classes from the core. 
</p>

<p>
The XML parsing handles two elements at parse time. One is the &lt;ref&gt;
element which is used for project references - that is relationships
between project files. The referenced project is parsed as well. The
second is the &lt;include&gt; element which includes either another complete
project or a project &lt;fragment&gt; directly into the project. All the other
elements are used to build a project model which is later processed in
the core.
</p>

<p>
The project model itself is organized like this
</p>

<p>
<ul>
  <li>A project contains</li>
    <ul>
      <li>named references to other projects</li>
      <li>targets</li>
      <li>build elements (tasks, type instances)</li>
    </ul>
    
  <li>A target contains</li>
    <ul>
      <li>build elements (tasks, type instances)</li>
    </ul>
 

  <li>A build element contains</li>
    <ul>
      <li>build elements (nested elements)</li>
    </ul>
</ul>
</p>
 
<p>
So, for now the project model contains top level tasks and type
instances. I'm still thinking about those and property scoping
especially in the face of project refs and property overrides. Anyway,
the running of these tasks is currently disabled.
</p>
 
<p>
Once the model is built, the commandline creates an execution manager
instance, passing it the initConfig built by Main.jar. It adds build
listeners and then starts the build using the services of the
ExecutionManager.
</p>
</section> 

<section name="Ant Libraries">
<p>
Before we get into execution proper, I'll deal with the structure of an
ant library and how it works. An antlibrary is a jar file with a library
descriptor located in META-INF/antlib.xml. This defines what
typedefs/taskdefs/converters the library makes available to Ant. The
classes or at least some of the classes for the library will normally be
available in the jar. The descriptor looks like this (I'll provide two
examples here)
</p>

<p>
<pre>
&lt;antlib libid=&quot;ant.io&quot; 
        home=&quot;http://jakarta.apache.org/ant&quot;
        isolated=&quot;true&quot;&gt;
  &lt;typedef name=&quot;thread&quot; classname=&quot;java.lang.Thread&quot;/&gt;
  &lt;taskdef name=&quot;echo&quot; classname=&quot;org.apache.ant.taskdef.io.Echo&quot;/&gt;

  &lt;converter classname=&quot;org.apache.ant.taskdef.io.FileConverter&quot;/&gt;
&lt;/antlib&gt;

&lt;antlib libid=&quot;ant.file&quot; 
        home=&quot;http://jakarta.apache.org/ant&quot;
        reqxml=&quot;true&quot; reqtools=&quot;true&quot; extends=&quot;ant.io&quot;
        isolated=&quot;true&quot;&gt;
  &lt;taskdef name=&quot;copy&quot; classname=&quot;org.apache.ant.file.copy&quot;/&gt;
&lt;/antlib&gt;        
</pre>
</p>

<p>
the &quot;libid&quot; attribute is used to globally identify a library. It is used
in Ant to pick which tasks you want to make available to a build file.
As the number of tasks available goes up, this is used to prevent name
collisions, etc. The name is constructed similarly to a Java package name -
i.e Reverse DNS order.
</p>

<p>
The &quot;home&quot; attribute is a bit of fluff unused by mutant to allow tools
to manage libraries and update them etc. More thought could go into
this.
</p>

<p>
&quot;reqxml&quot; allows a library to say that it wants to use Ant's XML parser
classes. Note that these will be coming from the library's classloader
so they will not, in fact, be the same runtime classes as used in Ant's core, 
but it saves tasks packaging their own XML parsers. 
</p>

<p>
&quot;reqtools&quot; allows a library to specify that it uses classes from Sun's
tools.jar file. Again, if tools.jar is available it will be added to the
list of classes in the library's classloader
</p>

<p>
&quot;extends&quot; allows for a single &quot;inheritance&quot; style relationship between
libraries. I'm not sure how useful this may be yet but it seems
important for accessing common custom types. It basically translates
into the class loader for this library using the one identified in
extends as its parent. 
</p>

<p>
&quot;isolate&quot; specifies that each task created from this libary comes from
its own classloader. This can be used with tasks derived from Java
applications which have static initialisers. This used to be an issue
with the Anakia task, for example. Similarly it could be used to ensure that
tool.jar classes are unloaded to stop memory leaks. Again this is
experimental so may not prove ultimately useful.
</p>

<p>
The &lt;typedef&gt; in the example creates a &lt;thread&gt; type. That is just a bit of fun which
I'll use in an example later. It does show the typedefing of a type from
outside the ant library however.
</p>

<p>
&lt;taskdef&gt; is pretty obvious. It identifies a taskname with a class from
the library. The import task, which I have not yet implemented will
allow this name to be aliased - something like
</p>

<p>
&lt;import libid=&quot;ant.file&quot; task=&quot;echo&quot; alias=&quot;antecho&quot;/&gt;
</p>

<p>
Tasks are not made available automatically. The build file must state
which tasks it wants to use using an &lt;import&gt; task. This is similar to
Java's import statement. Similarly classes whose ids start with &quot;ant.&quot;
are fully imported at the start of execution.
</p>
</section>

<section name="Mutant Configuration">
<p>
When mutant starts execution, it reads in a config file. Actually it
attempts to read two files, one from $ANT_HOME/conf/antconfig.xml and
another from $HOME/.ant/antconfig.xml. Others could be added even
specified in the command line. These config files are used to provide
two things - libpaths and task dirs.
</p>

<p>
Taskdirs are locations to search for additional ant libraries. As people
bundle Ant tasks and types with their products, it will not be practical
to bundle all this into ANT_HOME/lib. These additional dirs are scanned
for ant libraries. All .zip/.jar/.tsk files which contain the
META-INF/antlib.xml file will be processed. 
</p>

<p>
Sometimes, of course, the tasks and the libraries upon which they depend
are not produced by the same people. It is not feasible to go in and
edit manifests to connect the ant library with its required support
jars, so the libpath element in the config file is used to specify
additional paths to be added to a library's classloader. An example
config would be 
</p>

<p>
<pre>
&lt;antconfig&gt;
  &lt;libpath libid=&quot;ant.file&quot; path=&quot;fubar&quot;/&gt;
  &lt;libpath libid=&quot;ant.file&quot; url=&quot;http://fubar&quot;/&gt;
&lt;/antconfig&gt;
</pre>
</p>

<p>
Obviously other information can be added to the config - standard
property values, compiler prefs, etc. I haven't done that yet. User
level config override system level configs.
</p>

<p>
So, when a ant library creates a classloader, it will take a number of
URLS. One is the task library itself, the XML parser classes if
requested, the tools.jar if requested, and any additional libraries
specified in the &lt;antconfig&gt;. The parent loader is the common loader
from the initconfig. unless this library is an extending library.
</p>
</section>

<section name="Mutant Execution">
<p>
Execution of a build is provided by the core through two key classes.
One if the ExecutionManager and the other is the ExecutionFrame. An
execution frame is created for each project in the project model
hierarchy. It represents the execution state of the project - data
values, imported tasks, typedefs, taskdefs, etc.
</p>

<p>
The ExecutionManager begins by reading configs, searching for ant
libraries, configuring and appending any additional paths, etc. It then
creates a root ExecutionFrame which represents the root project. when a
build is commenced, the project model is validated and then passed to
the ExecutionFrame.
</p>

<p>
the ExecutionFrame is the main execution class. When it is created it
imports all ant libraries with ids that start with ant.*. All others are
available but must be explicitly imported with &lt;import&gt; tasks. When the
project is passed in, ExecutionFrames are created for any referenced
projects. This builds an ExecutionFrame hierarchy which parallels the
project hierarchy. Each &lt;ref&gt; uses a name to identify the referenced
project. All property and target references use these reference names to
identify the particular frame that hold the data. As an example, look at
this build file
</p>

<p>
<pre>
&lt;project default=&quot;test&quot; basedir=&quot;..&quot; doc:Hello=&quot;true&quot;&gt;

  &lt;ref project=&quot;test.ant&quot; name=&quot;reftest&quot;/&gt;

  &lt;target name=&quot;test&quot; depends=&quot;reftest:test2&quot;&gt;
    &lt;echo message=&quot;hello&quot;/&gt;
  &lt;/target&gt;

&lt;/project&gt;
</pre>
</p>

<p>
Notice the depends reference to the test2 target in the test.ant project
file. I am still using the &quot;:&quot; as a separator for refs. It doesn't
collide with XML namespaces so that should be OK.
</p>

<p>
Execution proceeds by determining the targets in the various frames
which need to be executed. The appropriate frame is requested to execute
the target's tasks and type instances. The imports for the frame are
consulted to determine what is the approrpiate library and class from
that library. A classloader is fetched, the class is instantiated,
introspected and then configured from the corresponding part of the
project model. Ant 1.x's IntrospectionHelper has been split into two -
the ClassIntrospector and the Reflector. When the task is being
configured, the context classloader is set. Similarly it is set when the
task is being executed. Types are handled similarly. When a type in
instantiated or a task executed, and they support the appropriate
interface, they will be passed a context through which they can access
the services of the core. Currently the context is an interface although
I have wondered if an abstract class may be better to handle expansion
of the services available over time.
</p>
</section>

<section name="Introspection and Polymorphism">
<p>
Introspection is not a lot different from Ant 1.x. After some thought I
have dropped the createXXX method to allow for polymorphic type support, discussed
below. setXXX methods, coupled with an approriate string to
type converter are used for attributes. addXXX methods are used for
nested elements. All of the value setting has been moved to a Reflector
object. Object creation for addXXX methods is no longer provided in the
reflector class, just the storage of the value. This allows support for
add methods defined in terms of interfaces. For example, the hacked Echo
task I am using has this definition
</p>

<p>
<pre>
    /**
     * testing
     *
     * @param runnable testing
     */
    public void addRun(Runnable runnable) {
        log(&quot;Adding runnable of type &quot;
             + runnable.getClass().getName(), MessageLevel.MSG_WARN);
    }
</pre>
</p>

<p>
So when mutant encounteres a nested element it does the following checks
</p>

<p>
Is the value specified by reference?
</p>

<p>
&lt;run ant:refid=&quot;test&quot;/&gt;
</p>

<p>
Is it specified by as a polymorphic type?
</p>

<p>
&lt;run ant:type=&quot;thread&quot;/&gt;
</p>

<p>
or is it just a normal run o' the mill nested element, which is
instantiated by a zero arg constructor. 
</p>

<p>
Note the use of the ant namespace for the metadata. In essence the
nested element name &lt;run&gt; identifies the add method to be used, while
the refId or type elements specify the actual instance or type to be
used. The ant:type identifies an Ant datatype to be instantiated. If
neither is specified, the type that is expected by the identified
method, addRun in this case, is used to create an instance. In this case
that would fail. 
</p>

<p>
Polymorphism, coupled with typedefs is one way, and a good way IMHO, of
solving the extensibility of tasks such as ejbjar.
</p>

<p>
OK, that is about the size of it. Let me finish with two complete build
files and the result of running mutant on them.
</p>

<h3>build.ant</h3>
<p>
<pre>
&lt;project default=&quot;test&quot; basedir=&quot;..&quot; doc:Hello=&quot;true&quot;&gt;

  &lt;ref project=&quot;test.ant&quot; name=&quot;reftest&quot;/&gt; 

  &lt;target name=&quot;test&quot; depends=&quot;reftest:test2&quot;&gt;
    &lt;echo message=&quot;hello&quot;/&gt;
  &lt;/target&gt;

&lt;/project&gt;
</pre>
</p>

<h3>test.ant</h3>
<p>
<pre>
&lt;project default=&quot;test&quot; basedir=&quot;.&quot; doc:Hello=&quot;true&quot;&gt;
  &lt;target name=&quot;test2&quot;&gt;
    &lt;thread ant:id=&quot;testit&quot;/&gt;
    &lt;echo message=&quot;hello2&quot;&gt;
        &lt;run ant:refid=&quot;testit&quot;&gt;       
        &lt;/run&gt;
    &lt;/echo&gt;

    &lt;echo message=&quot;hello3&quot;&gt;
        &lt;run ant:type=&quot;thread&quot;&gt;       
        &lt;/run&gt;
    &lt;/echo&gt;
  &lt;/target&gt;

&lt;/project&gt;
</pre>
</p>

<p>
If I run mutant via a simple script which has just one line
</p>

<p>
java -jar /home/conor/dev/mutant/dist/lib/start.jar $*
</p>

<p>
I get this
</p>

<p>
<pre>
test2:
     [echo] Adding runnable of type java.lang.Thread
     [echo] hello2
     [echo] Adding runnable of type java.lang.Thread
     [echo] hello3

test:
     [echo] hello

BUILD SUCCESSFUL

Total time: 0 seconds
</pre>
</p>

<p>
Lets change the &lt;run&gt; definition to 
</p>

<p>
&lt;run/&gt; in test.ant and the result becomes
</p>

<p>
<pre>
test2:
     [echo] Adding runnable of type java.lang.Thread
     [echo] hello2

BUILD FAILED

/home/conor/dev/mutant/test/test.ant:10: 
No element can be created for nested element &lt;run&gt;. 
Please provide a value by reference or specify the value type
</pre>
</p>


</section>
</body>
</document>