Introduction to NetKernel
-------------------------

by Brian Sletten (brian@bosatsu.net)

This tutorial is licensed unded the Creative Commons Attribution 2.5 license (http://creativecommons.org/licenses/by/2.5/legalcode).

Feedback is encouraged.


Getting NetKernel and the Basics:
---------------------------------

- Go get NetKernel: http://www.1060.org/download/
- java -jar <jarfile> to <install>
- We'll refer to the installation directory as <install>
- Fetch http://www.bosatsu.net/talks/examples/NetKernel-Examples.zip
- unzip this into <install>
- verify that this happened correctly by the presence of a modules/workbench/nfjs directory under <install>
- start NetKernel on the CL with bin/startup.bat or bin/start.sh
- hit the frontend fulcrum at http://localhost:8080
- hit the backend fulcrum at http://localhost:1060

Note: the fulcrums will be explained below.

(if you are curious about the magic number "1060" Roman(X + M + L ) = DEC( 1060 ) )

Also note, I highly recommend using a browser that does the right thing with XML (IE, Firefox, etc. Safari doesn't often do the right thing) for the demos below. The examples are based on my NFJS NetKernel talk.

Configuring the Front End Fulcrum:
----------------------------------
The frontend fulcrum "pivots" client requests into NetKernel URI space. The backend fulcrum does the same thing, but is designed for hitting the control panel, documentation, etc.

If at this point, you want to change the port that the frontend fulcrum listens on to avoid conflict with other Java web apps:

- edit modules/mod-fulcrum-frontend/etc/TransportJettyConfig.xml; search for 8080 and set it to whatever you like; restart NK; if you do change this, all of the examples below will need to be tweaked to work.

Summary:
--------

NetKernel is built out of modules that deploy applications, support for transports (mapping external requests into NK URI space: HTTP, JMS, SMTP, etc.), documentation and all the cool NK concepts of accessors, representations, aspects (immutable views, not AOP aspects) and transreptors (things that are registered with the kernel to transform the structural form of an aspect; think String to JDOM, JDOM to ByteArray, etc.). The microkernel maps requests into accessors based on their URIs. Internally, NK uses an active URI scheme that they are trying to standardize via the IETF.

More on Active URIs here (if NK is running):

http://localhost:1060/book/solutiondeveloperguide/doc_guide_activeURI

We'll mostly be using active:beanshell, active:xquery and active:dpml, but there are tons of schemes and you can add to them if you want.

NetKernel is only partially about XML processing, that is really just the first vertical market they went after using the generalized architecture of URI-based resource manipulation.

Using the Workbench Module:
---------------------------

Any real development you'd do would be as modules, but we'll start with a special one called the workbench. This is a module that checks the filesystem for changes in order to hotswap the resource/scripts/etc.

Look under <install>/modules/workbench. The directories there map to the first level after workbench in the URL to access it.

http://localhost:8080/workbench/nfjs/basic/hello.xml

If NK is running and you go here, you should be served up the static file in the directory <install>/modules/workbench/nfjs/basic. We'll talk more about the URI rewriting in a bit.

To execute our first piece of functional NetKernel, we'll hit a DPML script which is an XML-based declarative scripting language:

http://localhost:8080/workbench/nfjs/basic/hellonfjs.idoc

If you look at this file, you'll see it isn't all that interesting, but it starts us up by showing us how to work with static resources:

<idoc>
	<seq>
		<instr>
			<type>source</type>
			<uri>ffcpl:/nfjs/docs/nfjs.xml</uri>
			<target>this:response</target>
		</instr>
	</seq>
</idoc>

DPML scripts are usually sequences of instructions that manipulate XML documents. In this case, we are "source"ing a static resource and setting it as the response to the execution of this script. The net effect is the same as directly referencing the static resource, but you may not want to expose direct access or may wish to manipulate it somehow.

This is a common pattern/naming convention for these scripts. ffcpl is a URI named from the strategy of looking to the filesystem first, classpath later. You can investigate it more, but it is just a scheme to allow local customizations to override what is in jar files for instance. Note that the ffcpl reference is relative to the top of the module directory structure which, in this case, is <install>/modules/workbench.

Let's do something more interesting.

Look under <install>/modules/workbench/nfjs/quotations. There is a file there called "q.xml" which has a bunch of quotations that we'll want to serve up in various ways.

Look in "authors.xq" in the same directory. This is an XQuery script that calls the XQuery doc function to turn an external resource (in this case a file) into a node and then does an XPath selection of all the authors. It repackages the results in an element called "authors".

Look in "quotations.bsh" in this directory. This is a BeanShell (http://www.beanshell.org/) script which calls the XQuery function in "authors.xq" and returns the results. This is a similar pattern to the DPML script we saw before, it just uses different syntax/underlying technologies.

void main()
{
	//Extract all speeches with lines containing the parameter argument
    req=context.createSubRequest();
    req.setURI("active:xquery");
    req.addArgument("operator","authors.xq");
    output=context.issueSubRequest(req);

    //Finally return response
    response=context.createResponseFrom(output);
    context.setResponse(response);
}

Detour note: You'll see this pattern over and over again. The context object is setup by the NK script engine environment to expose the context of the request. This script will be compiled into Java bytecode once (unless it changes). Any objects created in this script are created new each time. This approach does not expose the lifetime of the module directly, so it is difficult to do initialization or maintain things statically like this. If you need to do that, you'll want to create your own modules.

First we set up the subrequest. This will be a synchronous call to the subsequent (active) URI. The request will be mapped to the appropriate module and scheduled on a kernel thread as one is available. The results are captured in the output variable which is used to generate the response for calling the script.

To access this, you would do this:

http://localhost:8080/workbench/nfjs/quotations/quotations.bsh

You should see the results wrapped in some other XML nonsense.

Detour note: Because everything is URI-addressable and RESTful, all context is passed as part of the request and the intermediate results (aspects) are immutable views of the underlying representations. They are also all candidates for caching. In the above example, the results of sourcing "q.xml" and selecting out the authors via "authors.xq" are both separately cacheable. The cache can be manipulated by indicating things such as "how painful is it to recreate this". You can investigate the cache here:

http://localhost:1060/ep+name@app_ext_introspect_cache 

If we want to query for specific authors, we might do something like we did in "abbey.xq":

<abbeysayings>
{
	for $author in doc("active:xquery+operator@ffcpl:/nfjs/quotations/authors.xq")//*
		where $author/name/last = 'Abbey'
		return $author/quotation
}
</abbeysayings>

This of course translates to building a node from the processing results of calling an active:xquery URI (in fact, our "authors.xq") and selecting off any nodes where the $author/last/name is equal to Abbey.

Again, we want to expose this through a BeanShell script, so we do so in "abbey.bsh" in the same directory:

void main()
{
	//Extract all speeches with lines containing the parameter argument
    req=context.createSubRequest();
    req.setURI("active:xquery");
    req.addArgument("operator","abbey.xq");
    output=context.issueSubRequest(req);

    //Finally return response
    response=context.createResponseFrom(output);
    context.setResponse(response);
}

You can run this by hitting this URI:

http://localhost:8080/workbench/nfjs/quotations/abbey.bsh

The processing flow is, of course:

abbey.bsh->abbey.xq->authors.xq->source(q.xml)

This begins to highlight the Unix-pipe like nature of NetKernel where you can gain a great amount of reuse by chaining together smaller tools that do one thing and do it well.

The downside of our approach is that we've hardcoded both the source of the quotations and the fact that we are looking for quotations from Edward Abbey (or anyone with the same last name).

We begin to solve this in <install>/modules/workbench/nfjs/quotations1 directory.

Our new XQuery to select the authors is in "authors2.xq":

declare variable $input as node() external;

<authors>
{
	$input//author
}
</authors>

Here we assume the pre-existence of a variable node called input from which we select the author elements. No more hard-coded quotation source.

We have a DPML script (just 'cause) to expose this new pipeline in "pipeline.idoc".

The first step calls "authors2.xq" with a different quotation source like this:

<instr>
	<type>xquery</type>
	<operator>authors2.xq</operator>
	<input>ffcpl:/nfjs/quotations1/q2.xml</input>
	<target>var:pipeline</target>
</instr>

Notice the node called "input" which matches our expected node in "authors2.xq". The results are captured in a variable called pipeline.

The next step in our new pipeline still hardcodes the author selection, but uses a different one:

<instr>
	<type>xquery</type>
	<operator>socrates.xq</operator>
	<input>var:pipeline</input>
	<target>var:pipeline</target>
</instr>

"socrates.xq" looks like this:

declare variable $input as node() external;

<socraticsayings>
{
	for $author in $input//*
		where $author/name/last = 'Socrates'
		return $author/quotation
}
</socraticsayings>

Fans of the movie "Real Genius" will be amused by hitting this URL to access this pipeline:

http://localhost:8080/workbench/nfjs/quotations1/pipeline.idoc

Of course, we still want to avoid this hardcode. "authorselect.xq" does this:
declare variable $input as node() external;
declare variable $param as node() external;

<authorsayings>
{
	for $author in $input//*
		where $author/name/last = $param/nvp/source
		return $author/quotation
}
</authorsayings>

We select all nodes from the $input node set where the last name/last element matches the element in the $param node. It is a NetKernel convention to pass name value pairs through XML snippets like this:

<nvp>
   <name>value</name>
   <name1>value1</name1>
</nvp>

We access this new pipeline through "pipeline2.bsh" (just cause). We will use our original quotation source and call our new doubly-parameterized "authorselect.xq".

void main()
{
	req=context.createSubRequest();
	req.setURI("active:xquery");
	req.addArgument( "operator", "authors2.xq" );
	req.addArgument( "input", "ffcpl:/nfjs/quotations/q.xml" );
	output = context.issueSubRequest( req );
	
	param=null;
	try
	{   
		param=context.source("this:param:param");
		
	} catch(Exception e)
	{   //Use default
	    param=context.source("defaultParam.xml");
	}
	
	req=context.createSubRequest();
	req.setURI("active:xquery");
	req.addArgument( "operator", "authorselect.xq" );
	req.addArgument( "input", output );
	req.addArgument( "param", param );
	output=context.issueSubRequest(req);
	
	response=context.createResponseFrom(output);
	context.setResponse(response);
}

You'll note that we want to pass on any parameters that are passed in by attempting to source as an XML document a parameter called "param". If there is no such one, we'll source "defaultParam.xml" which looks like this:

<nvp>
	<source>Flaubert</source>
</nvp>

You can access the new pipeline this way:

http://localhost:8080/workbench/nfjs/quotations1/pipeline2.bsh

and learn an important lesson about life in the process compliments of Mssr. Flaubert.

We also can "inline" things like the parameter at various stages like we do in "pipeline2.idoc":


<idoc>
	<seq>
		<instr>
			<type>xquery</type>
			<operator>authors2.xq</operator>
			<input>ffcpl:/nfjs/quotations/q.xml</input>
			<target>var:pipeline</target>
		</instr>
		
		<!-- NOTE THE INLINED XML SNIPPET WHICH GETS COPIED
			 INTO A VARIABLE CALLED "param" BEFORE WE CALL
			 "authorselect.xq" -->
			
		<instr>
			<type>copy</type>
			<operand>
				<nvp>
					<source>Abedi</source>
				</nvp>
			</operand>
			<target>var:param</target>
		</instr>
		
		<instr>
			<type>xquery</type>
			<operator>authorselect.xq</operator>
			<param>var:param</param>
			<input>var:pipeline</input>
			<target>var:pipeline</target>
		</instr> 		
		<instr>
			<type>copy</type>
			<operand>var:pipeline</operand>
			<target>this:response</target>
		</instr>
	</seq>
</idoc>

We can access this through this URL:

http://localhost:8080/workbench/nfjs/quotations1/pipeline2.idoc

Calling NetKernel from external Java code:
------------------------------------------

Using Apache's HttpClient (which requires Commons Logging and Commons Codec), you might access this functionality in Java as follows (we pass in a different author as a parameter):

package net.bosatsu.nk;

import java.io.ByteArrayOutputStream;
import java.io.InputStream;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.GetMethod;

public class NKClient {

	public static void main(String[] args) {
		try {
			HttpClient client = new HttpClient();
			GetMethod gm = new GetMethod(
					"http://localhost:8080/workbench/nfjs/quotations1/pipeline2.bsh?author=Abedi");

			int result = client.executeMethod(gm);

			InputStream is = gm.getResponseBodyAsStream();

			ByteArrayOutputStream baos = new ByteArrayOutputStream();

			byte[] buffer = new byte[2048];
			int numRead = 0;
			while ((numRead = is.read(buffer)) >= 0) {
				if (numRead > 0) {
					baos.write(buffer, 0, numRead);
				}
			}

			baos.close();
			is.close();

			System.out.println("Results:\n" + new String(baos.toByteArray()));
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}

This is even simpler using the Restlet (http://www.restlet.org) APIs:

package net.bosatsu.nk;

import java.io.IOException;

import org.restlet.Client;
import org.restlet.data.Protocol;

public class NKClient2 {
	public static void main( String [] args ) {
		Client client = new Client(Protocol.HTTP);
		
		try {
			client.get("http://localhost:8080/workbench/nfjs/quotations1/pipeline2.bsh?author=Abedi").getEntity().write(System.out);
		} catch( IOException ioe ) {
			ioe.printStackTrace();
		}
	}
}


Non-XML Results:
----------------

The RESTful interface and focus on resources does not require us to use/return only XML. NetKernel ships with examples of generating audio files with Java sound APIs:

http://localhost:8080/workbench/example_beanshell/DynamicSound.bsh

as well as generating PNGs representing Hilbert spaces:

http://localhost:8080/workbench/example_beanshell/HilbertSpaceSVG.bsh

Your browser should figure out how to consume the results based on the MIME types.

We have a few examples that show rendering SVG files into PNGs and then calling Java code to manipulate the rendered image:

http://localhost:8080/workbench/nfjs/svgimage.bsh
http://localhost:8080/workbench/nfjs/svgimage.idoc

The point is simply to stress that NetKernel's elegant abstractions, caching and scalability characteristics are useful beyond the scope of just processing XML.

Other Languages:
----------------
Before we move on to modules, notice that under the workbench there are some examples written in other languages. You'll note the similarity between each of them.

Groovy (<install>/modules/workbench/example_groovy):

req = context.createSubRequest();
req.setURI("active:xslt");
uris = [ "operator" : "../support/resources/act_to_html.xsl",
         "operand" : "../support/resources/lear.xml" ]
uris.each { | key, value | req.addArgument(key,value) }
result = context.issueSubRequest(req);
resp = context.createResponseFrom(result);
context.setResponse(resp);

and JavaScript (<install>/modules/workbench/example_javascript):

//Create and issue XSLT request
sr=context.createSubRequest();
sr.setURI("active:xslt");
sr.addArgument("operand","../support/resources/lear.xml");
sr.addArgument("operator","../support/resources/act_to_html.xsl");
rep=context.issueSubRequest(sr);
    
//Create response
resp=context.createResponseFrom(rep);

//Issue response
context.setResponse(resp)

and Python (<install>/modules/workbench/example_python):

# Create and issue XSLT request
sr=context.createSubRequest();
sr.setURI("active:xslt");
sr.addArgument("operand","../support/resources/lear.xml");
sr.addArgument("operator","../support/resources/act_to_html.xsl");
rep=context.issueSubRequest(sr);

# Create response
resp=context.createResponseFrom(rep);

# Issue response
context.setResponse(resp);

As soon as JRuby is released, 1060 Research plans to support Ruby as well.

Modules:
--------
Modules are versioned units of deployment in NetKernel. They encapsulate behavior, public and private URI spaces, static resources, etc. NetKernel uses separate class loaders per module so it is possible to have multiple versions of things like Jar files in different modules at the same time. This makes it easy to migrate resource users to new versions of modules over time as needed.

You can see which modules are defined/deployed by hitting:

http://localhost:1060/ep+name@app_ext_introspect_modules

It may have surprised you so far that we have done so much XML without having to deal with conventional XML Jar files like SAX, DOM, JDOM, etc. This is because NetKernel has a module that contains these. You can find out more information about this core module by looking here.

http://localhost:1060/introspect/module+uri@urn:org:ten60:netkernel:ext:xml:core

The Workbench module has a dependency on this core XML module. You can tell this by looking at the workbench module definition file or looking at the RequestHandler import section here:

http://localhost:1060/introspect/module+uri@urn:org:ten60:workbench+version@version:1.1.0

There are a few key points to remember here. Modules export both URIs and classpaths. When the workbench module class loader needs to load any org.jdom.* class, NetKernel will look at the dependencies in the module definition and see that one of the imported modules offers to handle those. This eases the burden on you from having to track latest and greatest stable versions of these libraries. Of course, if you have a need for a different version of these libraries, you can handle the dependencies and class path loading yourself.

Similarly, when NetKernel looks for a module to handle one of our "active:dpml" or "active:xquery" URIs, it looks through the imported module definitions for a module that promises to handle that. In these cases, it would be the DPML engine module and the XQuery engine module respectively:

http://localhost:1060/introspect/module+uri@urn:org:ten60:netkernel:ext:dpml
http://localhost:1060/introspect/module+uri@urn:org:ten60:netkernel:ext:xquery

another thing to notice in the workbench definition is how URIs are exported and rewritten. You'll notice this line:

.*:/workbench/.*

This means that it will respond to any URI grounded at a workbench subelement.

It converts between the public and private URI spaces (logical vs. physical) with the following rewrite rule:

ffcpl:/workbench/(.*) -> ffcpl:/$1

other interesting rewrites include:

(.*)/ -> $1/index.html
(.*\.idoc)(.*) -> active:dpml+operand@$1$2
(.*\.bsh)(.*) -> active:beanshell+operator@$1$2
(.*\.py)(.*) -> active:python+operator@$1$2
(.*\.(groovy|gy))(.*) -> active:groovy+operator@$1$3
(.*\.js)(.*) -> active:javascript+operator@$1$2

If the user doesn't specifying any subresource, we return a documentation page (<install>/modules/workbench/index.html).

If they ask for a file that ends in one of those file types, they get rewritten to a URI that another module will handle.

If you're curious, the workbench module definition is in <install>/modules/workbench/module.xml

Information about module definitions can be found here:

http://localhost:1060/book/solutiondeveloperguide/doc_guide_module_develop

There is a sample module in the modules directory called "brian-sample". It is not deployed by default. Without going through the module management features of the backend fulcrum, you will have to modify <install>/etc/deployedModules.xml to let NetKernel know about it.

If you installed the examples, you should have a patched version of the deployedModules.xml file with this module commented out. Uncomment it. (Note that some modules are Jar'ed up, others are just directories. This is mostly a matter of choice/ease of deployment. You'll probably want to develop modules in unjar'ed form but jar them up for deployment.)

At this point, if you shut NetKernel down and restarted it, it would see the module definition but we wouldn't be able to access it directly. This is because we need to request that the frontend fulcrum "pivot" external HTTP requests into the internal URI space. To do that, we need to modify <install>modules/mod-fulcrum-frontend/module.xml. If you installed the examples, there should be a commented out import statement down after the "Add your modules below here..." section.

Uncomment that, bounce NetKernel (CTRL-C if you started it by hand on the CLI, then bin/start.sh or bin/startup.bat) and try to hit:

http://localhost:8080/brian/

You should see "Hello this is my Brian service."

If that doesn't work, look carefully to see if you left something partially commented.

Look through the <install>/modules/brian-sample/module.xml to see what we are exporting as far as URIs are concerned and which modules we import. Look at the .bsh and other files. You'll note that one is calling Java code that it finds relative to the module root. If you create a lib directory in you module and put Jar files in there, they will be available to your module. (Remember to import module definitions that already handle Jar files where possible.)

If you hit this URI, you'll see a secret message:

http://localhost:8080/brian/secret.bsh

What's interesting is how this came to be. Look through the code and you'll see a mixture of BeanShell, Java, DPML and references to the very cool simple tree manipulation (STM) features of NetKernel (more info: http://localhost:1060/book/solutiondeveloperguide/doc_tutorial_stm_by_example)

You can see a basic attempt at exposing a RESTful interface in this module. For the demo, we are delegating to another module, but if you type something like:

http://localhost:8080/fetch+url@http%3A//www.cnn.com

You'll see most of CNN's homepage show up. It's most because relative links are not pulled in, but this is just a simple example.

The brian-sample module exports .*:/fetch.* and rewrites things to point to the "fetch.bsh". This script finds out what URL is passed in through the url parameter (+url@<url> in active URI syntax) and then forwards the request to the http:get accessor.

Other Examples:
---------------

I've included a timezone module from the 1060 Research folks. You can hit it by going to:

http://localhost:8080/time/

You'll notice that we are quite able to describe RESTful APIs using English and HTML rather than WSDL!

To find out the current time in Hawaii, goto:

http://localhost:8080/time/timezone/US/Hawaii

To find out the current time in the EST timezone:

http://localhost:8080/time/timezone/EST

Pop Quiz:

What do you think the URL for finding out the time in the PST timezone might be?

You're probably correct, but you've just stumbled across a huge debate in REST community.

Tim Berners-Lee and the W3C TAG argue that you should not assume any semantic meaning to URIs. "Backing up" a level in the hierarchical part of the URI does not guarantee that you will find more content there. From their perspective (and for good reasons), they argue that you should treat a URI as a hash code that is simply dereferenced.

You will, however, find individuals who disagree with this perspective and believe that, as long as URIs can be *TREATED* as opaque hash values, then designing them to allow the kind of serendipitous discovery we just observed is a useful feature.

To see an example (although not a well-implemented one!) of wrapping an existing API as a RESTful service, you should hit this URL after the following installation instructions:

http://localhost:8080/workbench/nfjs/graph/graph.bsh

Because this involves an LGPL'ed API that I didn't want the burden of complying with, I don't distribute it. To run the above example, please go download a version of JFreeChart from:

http://www.jfree.org/jfreechart/

Once you extract the download, create a directory called lib under the modules/workbench directory and copy the jcommon-x.x.x.jar and jfreechart-x.x.x.jar into it. This will introduce these jar files to the workbench module.

Now the above example should work. If not, trying rebooting NetKernel.

You'll notice that graph.bsh returns an HTML page with a reference to an image that is dynamically generated by NetKernel. Entire portions of an HTML page can be efficiently generated by NetKernel. The img "src" attribute is a URL that points back into a NetKernel script that will render the graph for you.

Generating HTML as Strings/StringBuffers is bad form. Cool NetKernel kids use XRL. But that is another tutorial:

http://localhost:1060/book/solutiondeveloperguide/doc_xrl_eg_index


Advanced Features:
------------------

Asynchronous Processing:
------------------------
Asynch/Parallel processing is very easy in NetKernel. You can find out more here:

http://localhost:1060/book/tutorial/doc_map_Async

Throttling:
-----------
Particular URIs can be throttled for load/license-control reasons:
http://localhost:1060/book/developerreference/doc_ura_throttle

Security per URI:
-----------------
http://localhost:1060/book/solutiondeveloperguide/doc_ext_security_URIGateKeeper

Scalability:
------------
NetKernel's design allows it to scale nicely from small memory footprint VMs up to multi-CPU servers. A discussion of how it stays up and healthy in the face of increased load/requests can be found here:

http://localhost:1060/book/administratorguide/doc_whitepaper_scalability