AONS 2 Development

DocBook + Images != Fun

2007-10-02T13:26:00.000+10:00

I've been using DocBook for a little while now, and I have to say, I wish there was a better alternative (something which provides a nice base format for the creation of multiple formats). My latest niggle is working with images. Like most things with DocBook, the solution starts off only a little difficult, then gets a fair bit more complicated.

Before I go on, I realise a big part of my complaint is to do with XSLT: at the end of the day, XSLT really is just about transforming the XML, and not about building finished documents. Any yet I can't help but feel that whatever solution a person uses for document creation should encompass the whole shebang, from the core documents to the images they contain.

Back to the topic of conversation, DocBook and Images. Getting the initial block of XML needed for images isn't that hard, and here it is:


<mediaobject id="Registry Types">
        <imageobject role="fo">
            <imagedata format="SVG" fileref="RegistryTypes.svg">
        </imagedata>
        <imageobject role="html">
            <imagedata format="PNG" fileref="RegistryTypes.png">
        </imagedata>
    </imageobject>
</imageobject></mediaobject>

You'll notice that I've specified two formats, one for the PDF generation and one for the HTML generation. This represents the fact that scalable vector graphic support is still not as widely accepted as we'd like. So, now we've got to create multiple images to support two formats. I could have just used a raster image for the PDF, but due to the need for higher print quality (and also the fact that Magic Draw doesn't provide configurable dpi on it's rasters) I chose to do both formats.

Now comes the trick: the 'fileref' is not relevant to either the HTML XSLT transformation or the FO XSLT transformation, it's only relevant to either the final destination of the HTML or as I would deem incorrectly, the base directory of the fop Ant task.

For the HTML, what this means is that you have to define appropriate tasks to copy the images to a directory accessible to the final production. Not too difficult but worth remembering.

For the PDF generation, you need to ensure that the images are relative to the base path where the Ant script is run... so the above declaration should probably be changed to something like "images/RegistryTypes.svg" so the "fop" task can access the required images.

AONS, Rails and Other Integration Frameworks

2007-08-16T17:37:00.001+10:00

I think I'm one of the last people to try out Ruby on Rails. Well, I will be last if I ever do.

Rather then state my opinions on RoR (how quickly one picks up another acronym) and risk the ire of the agile community, I think I'll just examine the technology stack within the context of creating AONS.

First off, I don't know Ruby on Rails very well. I think I understand the principal though:
- Add Ruby, a development language which is geared towards rapid development
- Integrate (or create) any Ruby middleware technologies in a best of breed approach, essentially abstracting the developer away from this choice (saves time on tasks like integrating with one of many persistence implementations and learning it's quirks)
- Create good facades for any commonly used subsystems (authentication, notification?)

In principal, I admire RoR for integrating all those technologies because integrating technologies on a per/project basis is a pain (and as Martin Fowler says, it's a great domain specific application for writing web applications). As an example on AONS, it took about four weeks integrating and de-quirking Hibernate JPA (and finding a nice flavour), Compass/Lucene, Spring, JFreeChart, Quartz, Castor, ROME, a REST exposure methodology and Spring Web Flow (mainly in choosing one of many, many view technologies). Most of this was spent familiarizing myself with JPA and trying a few different implementations so realistically it could be cut down to one or two weeks if I had to repeat. But the point here is that if it is a common task, why do this at all? So, on this point, I do admire Ruby on Rails.

Where I disagree with Ruby on Rails in the long term is that learning and integrating with new technologies is an ability. Whilst it is nice not to have to constantly waste effort doing these kinds of tasks, learning various approaches to wire in different technology stacks is a must have skill for all but the most trivial projects. So, I think the bonus Ruby gives should only be that it removes most of the integration tasks from a project - but at the end of the day a good developer should always be willing and able to integrate should existing technologies not prove sufficient. I believe part of the attraction to Ruby on Rails is that at the end of the day, we're lazy creatures. That's not necessarily a bad thing: having 80% of the integration job done means you'll get quicker return on investment getting that last part of integration done. But the ability to integrate is something which needs to be constantly honed otherwise it atrophies. I also believe Ruby on Rails offered a nice foot in the door to a lot of developers with minimal integration experience. Suddenly they could compete with the Java gurus who wore their hard won knowledge of integration like a badge. But here's where I believe the crux is: nothing can make up from the experience of learning to wire in some misbegotten technology. Every successful developer will have to do it, and time will be a good measure to show whether those who took up RoR learned how to take steps beyond the initial RoR framework... Did they keep learn the "art" of integration or are they left waiting for the next RoR (SEAM?).

So, integrating the technologies is an all go - now creating subsystems. I think I do agree that for a given domain specific language, you need a set of domain specific subsystems. I think RoR's success is a testament to the lack of commonly used functionality within the J2EE/EJB core (again back to the idea Martin Fowler talks about with Java being a domain independent language). Yes, in terms of "enterprisey" features they (J2EE/EJB) are fantastic, but up until recently, most of those features were hard to implement/understand. People need facades for the domain their working in... the job should be easy if you're doing a task in a common way. It may be necessary to slip past the facade should a job be more complicated, but that option should be there to do something simply. The main success of projects like SEAM/RoR is to go, "Hey, Java developers, use us and accelerate your development". And again, I'm forced to tip my hat in their direction.

Just one point before I finish. The work I did for AONS could quick easily be stripped into an agile integration core offering similar features to something like SEAM or Ruby on Rails. Where it would fall down (without proper resourcing) would be the many, many hours it takes to write good examples and documentation so that using the framework became easy and intuitive.

So, that's my thinking on Ruby on Rails (and a touch on SEAM). I'm not sure how helpful this will be to anyone... if nothing else maybe it will just resonate with thoughts going through other developers minds about the current successes in the small to medium application development world.

Cheers,

Dave

Got 2 phase deployment done

2007-08-07T14:36:00.000+10:00

I've now successfully added a two phase option to the AONS build. This is a handy mechanism to get the build completed up to the point where it would require client specific information (database connection info) and then stop the build there. What this means is that someone downloading a "release" sees a much more lightweight project structure then someone working with the full development structure. Another benefit is that we don't need to ask the deployer to compile the code against any required server libraries which can't be included in the release (due to more restrictive licensing).

This release also contains the compiled DocBook documentation (PDF, HTML single and HTML split) generated from the source DocBook XML which is included in the main project. So far I've got a basic Install Guide written and am working on the As Build Specification.

I know there's a lot of room for improvement, but between writing new features, bug fixing and finishing off existing documentation it's still a bit overwhelming. If anyone is reading this and downloads the release build from the SourceForge, please let me know any issue encoutered or any "usability" ickiness which needs improvement. Any tweaking with included file structures is actually pretty easy to do, it's really just needs a bit of perspective from someone who hasn't had their face right against the coal face for the past few months.

Well, for what it's worth, the build release is over here. Remember, we're still in Beta so this build should be taken with a hefty grain of salt.

hit with the documentation stick

2007-08-01T17:46:00.001+10:00

Getting towards the end of this iteration of AONS, I'm starting to shift gear and work more on documentation. In an ideal world, I'd get the following done:

As Built Specification (design/implementation focused document)

How AONS works (methodologies, design patterns etc)
Developing AONS (code conventions, source control etc)
Building AONS (development build and two phase distribution build)
Installation Guide (may also be an independent document)

User/Admin Guide
REST Interface Guide
Service Usage Model (SUM)
Future Extension Guide (anything we'd really like to have implemented in AONS, which there's a lot, but just plain ran out of time)
Fix up any not completed Javadoc

As always, time is heavily against us... and documenting this stuff with the effort it deserves is a little daunting. Ah well, if I just spend a few hours every day until the end of the project, I should be able to knock over most of it.

From a technical perspective, one facet of this activity I am enjoying is that I'm using DocBook as the source format for my documentation. This required a little learning to get started, but the idea of modular documents really suits my way of thinking. There's nothing worse then having to repeat sections within documentation and knowing that they'll get out of synch with the first update.

One (minor) criticism with DocBook is the WSYIWYG tools are a little clunky. Not that I'm really using them much now that I've got my nice handful of tags under my belt. DocBook seems to be one of those things which you will just keep being surprised by it's breadth of features, but most tasks only require a very small subset.

Another (minor) concern with DocBook is the usage of the DTD. I was (and am) a little confused as to whether there is an XML Schema available for DocBook. The main reason I'd personally much prefer it is that the Eclipse content assist XML editor seems to only handle XML Schema, not DTD. I guess it comes down to DocBook being closer to SGML then XML so maybe they can't use it... but it's kind of bucking the trend with most tools preferring XML Schema. Ah well, if it aint broke...

Okay, seriously, last concern with DocBook. Xml Entity objects. They really are outdated and there are many better ways to reference other objects. It kind of feels like a poor mans Dependency Injection. Ant used to use them, but there were all sorts of issues... there must be a better way to reference other documents. My preference would be to leave the actual dependencies as undefined within the file which requests them (if they are resolved locally) and wait until all documents are composed together with a top level import document.

[Update: Okay, just found the documentation for DocBook with XIncludes... looks like XML Entities were thrown out.]

file identification funny

2007-07-11T11:11:00.000+10:00

Occasionally one stumbles upon a code comment which both accurately states what the program is doing and also brings a smile to the face of the reader. Whilst reading through the source code for the "file" utility in it's attempts to identify a file, I came across this one which I thought was quite good:

/*
* Try compression stuff
*/
 if (!zflag || __lf_zmagic(buf, nb) != 1)
  /*
   * try tests in /etc/magic (or surrogate magic file)
   */
  if (__lf_softmagic(buf, nb) != 1)
   /*
    * try known keywords, check for ascii-ness too.
    */
   if (__lf_ascmagic(buf, nb) != 1)
       /*
        * abandon hope, all ye who remain here
        */
       __lf_form("data");

People Forget

2007-07-05T17:49:00.000+10:00

After reading this about Spring I kind of got a little bit annoyed. The number of people who forget or don't even know how bad a job configuration management used to be before Spring gave it one hell of a kick in the pants is staggering. Also to be thanked are Martin Fowler and the other IoC containers for starting the push.

Just yesterday I found a lovely refresher on 'ye old days of bizarre configuration' in Lucene's SegmentReader via a global property which is still valid in it's current release. But nasty things like that don't happen much anymore do they? Now that the bar has been raised, isn't everyone quick to bemoan nasty things like well structured XML configuration.

Oh geez, and the major complaint that "how would we live without Spring?". Continuing on from that, why would you? I'd be willing to bet that if you're using Spring in a deployable Application and not a piece of middleware, you're going to need a lot more then a Spring jar to run your application, and guess what, it's going to be a small to large job to change any of them. That's the world we live in as integrators first and developers second. If a developer doesn't like it, don't use any middleware and see how long it takes.

Next, complaining about the start up time: again... is this really relevant? Not many people have weekly 'Friday server startup races'. Also, an empty Spring container will take a little while to start up, but once done, those load operations were concentrated in one tiny bit of the applications life: meaning that you don't keep doing them for every operation, saving time. See that last part there, that was key: do things which have to get done once, then they don't have to be done every time you want to do the action. That logic will work as long as you don't do any unnecessary load operations in the Spring load... but then I guess you can also use Lazy loading if you want. See, options... it's all about options.

And the Spring people are nice too... Yatzaa!!

lucene issue

2007-07-04T13:01:00.000+10:00

I feel sorry for Shay: he's written a beautiful object indexing layer, Compass, on top of Lucene (the equivalent of writing the first good Object Relational mapping tool but for searching), yet is let down by the underlying interface to Lucene. Don't get me wrong, Lucene is great, but it does have some quirks.

Take this one for example, setting a custom SegmentReader. Compass uses a custom SegmentReader (unsurprisingly called CompassSegmentReader), which in an of itself isn't a huge modification of Lucene. The quirk comes into play when we deploy multiple applications into the same Servlet container. It gets back to the method of setting the custom SegmentReader in Lucene; not changing a property on a factory or another configuration parameter... no, Lucene needs to set a system wide property. Yuck. This means that when Compass set's it's custom SegmentReader, it sets it for everything in that JVM. So App1 using Compass is okay, but App2 using pure Lucene then falls over unless the Compass jar file is included. Here's a thread on the topic over at the Compass forums.

In my case I've found this out when deploying DSpace on the same application server as AONS. I'm not sure how I'll get around it: I can probably ask the DSpace developers to include Compass in their distribution, or I'll probably just put in a README note about it. Either solution is less then ideal but probably the least path of resistance by far. Maybe the Lucene developers would be willing to change this... but the original post about this problem in the Compass forums was in 2005 so I have my doubts.

Well, here's the offending code from org.apache.lucene.index.SegmentReader (valid as of Lucene 2.2.0):

static {...

try {

String name = System.getProperty("org.apache.lucene.SegmentReader.class", SegmentReader.class.getName());

IMPL = Class.forName(name);

} catch (ClassNotFoundException e) {

throw new RuntimeException("cannot load SegmentReader class: " + e, e);

} catch (SecurityException se) {

try {

IMPL = Class.forName(SegmentReader.class.getName());

} catch (ClassNotFoundException e) {

throw new RuntimeException("cannot load default SegmentReader class: " + e, e);

}

}

}

This kind of code is exactly why Inversion of Control (aka Dependency Injection) was so good when it came along: it got rid of static calls and global system properties like this and made property injection obvious and intuitive. I'm sure when this segment of code was first written, it was completely acceptable to do things like this... but nowadays configuration and anticipated customisation should be a little bit more elegant.

And now ... Hibernate's JPA

2007-06-14T17:01:00.000+10:00

So, with a bit of a hit and miss experience with OpenJPA after Toplink JPA, I decided to give Hibernate JPA (Hibernate Entity Manager) a go before going back to Toplink JPA with my tail between my legs. My initial reservation for trying out Hibernate's JPA implementation was mainly due to the rift between the Spring/Interface 21 people and the Hibernate/JBoss people. Whilst Hibernate and Spring have been and remain a fantastic combination, the bitter blood between the too camps can be a little intimidating. But... with another glowing article on using Spring and Hibernate JPA in combination, I set about my task.

Well, only and hour and a half later, and I can say that my faith in JPA has been restored, along with a nice bonus that I can remove any reference to a javaagent: Hibernate's implementation needs no sort of special run time weaver, which makes me wonder why the others do.

Looking back at my experiences with OpenJPA and Toplink, I'm willing to bet most of the pain I endured was due to the special weavers needed to run them on servlet containers like Tomcat. The special weavers bring out some very confusing bugs with IllegalAccessErrors, often quite deep in the Java reflection stack. So, depending on where you're going to be deploying to, I'd offer the following advice (with the proviso that this advice is only based on observations and not hard fact):

lightweight servlet container like tomcat: maybe avoid OpenJPA and Toplink until the javaagent weavers are no longer necessary, try Hibernate (or Kodo if it's a commercial application)
EJB 3 container: The weaver concerns do not apply, OpenJPA and Toplink should be fine in these environments

[Note: I haven't actually had any experience on a full blown EJB 3 server as of yet with OpenJPA or Toplink, so feel free to let me know if your own experiences differ and they don't work as expected.]

Tried using OpenJPA

2007-06-14T14:36:00.000+10:00

Toplink gave me yet another strange bug... which I wasn't even game to try and figure out (the object graph parsing and object and somehow updating it's ID invalidly... on a find operation).

I used to be a developer who'd happily hunt bugs whilst time slipped away. Since then I've realised that unless the bug is relatively easy to find/fix, the time taken away from a project is important and not really worth it. So, I through my hands up in disgust and decided to try test out JPA's benefits of being able to easily swap in/out a different implementation.

Oh how naive I was. I tried OpenJPA. Now, don't get me wrong, I think it's good that there will be another implementation of JPA out there - I just didn't feel OpenJPA was ready for prime time. Yes, they have "graduated" from their incubator status, but I could not get it working. Maybe Toplink let me do some naughty annotation things which OpenJPA didn't... that would have been fine if true. What seemed to be my major issues with it were:

Weaving
Working with Compass
Weaving (again)
Weaving (yes, again)

Now, I had a few heartaches getting compass to work with Toplink, but I found this doubly so with OpenJPA. I also found the documentation on utilising the javaagent to be sparse and outdated. When I tried OpenJPA's weaver/enhancer I could not get past the myriad of NoClassDefFound errors - in my mind, adding a javaagent jar file is okay for a temporary hack, but having to add another six libraries to the classpath is just those few steps too far towards making an almost impossible deploy job for a War file.

Lastly, and this will probably be the reason I abandon OpenJPA: I could not get past the IllegalAccessException's with the "unsafe method invocation" and "GeneratedConstructor${index} could not access superclass".

I don't completely understand the IllegalAccessException error: it appeared to be coming from the Java annotation/reflection stack. So maybe it was Spring's weaver, maybe it was compass, maybe it was OpenJPA... I really don't know. More importantly, I don't care. The level of complexity totally removed all the benefits of using JPA. So I'm going back to Toplink, with my tail between my legs - and will put up with one serious workaround in comparison to the alternative of using OpenJPA.

I'd like to see OpenJPA become more usable for those using Tomcat - but it definitely isn't there yet.

Eclipse 3.3rc1 and imports

2007-06-05T16:14:00.001+10:00

I'm using Eclipse 3.3 rc 1 and the automatic insertion of imports at the top of Java files isn't always working properly. I just had to hand craft my first import in what must be years. So I'll just take this opportunity to say thank you to Eclipse and all those other lovely IDE's out there which have made my daily development life so much easier - and even if it isn't working now, I'm sure it'll be working for the final.

I'm not going to bother creating a formal bug for this though - I'd have to first log on to the Eclipse site, which would mean username and password retrieval as well as first searching if the bug is even there already then lodging it if not. I should lodge a bug report, but this time I'm not. I'm like a rebel without a cause;P

Okay I caved... the bug report is over here. So much for being a rebel;P

[Note: This bug was pretty quickly closed as "works for me" since I didn't have a solid set of steps to reproduce the bug. Whilst I'll agree with this kind of thing in the most case, if it is stated that the bug is happening intermittently there is a good chance that there will never be a set of steps to reproduce this bug. Well, a bug that is happening only 1% of the time on a feature that is used 1000 times a day by a developer will very quickly come back to haunt when the latest version of Eclipse goes out for official 3.3 release. Intermittent bugs should only ever be ignored if they occur on a function which is used rarely.]

Compass, Cascade and Cyclic References (oh my)

2007-06-05T10:59:00.000+10:00

As identified here, I'm having another issue with Compass. Nothing major, just cyclic references between objects and cascades are a little wacky.

I use cyclic references between formats to link one format to another via a previous/next relationship. I am hoping to utilise Compass to do indirect matching on objects, but may have to hold off for a while the above mentioned issue gets resolved. It's only been a day though so plenty of time for a response from Shay or another talented compass developer.

If I can't get resolution on this issue, I could match indirect matches via a post-search process which just retrieves formats found in either a previous/next relationship. The problem I have with that is then my search mechanisms aren't as general purpose in the application... I'd need to somehow wire in post processors which would depend on the type of CompassHit found. I could do it, but it would be pretty ugly and may be easier to just wait for the outcome of my forum post.

Sometimes it's hard to walk away from a feature which is "almost there", but I think in this case it's better that I leave a footnote here and look back at this in a month or so.

[Update: it appears my original idea about Compass' usage of SearchableReferences was a little off the mark. I figured that SearchableReferences were like pseudo properties; they would allow an object to match search criteria even though they didn't have the search match directly on them. This isn't the case... they're more a convenient way of storing object graphs.]

Using Eclipse's "go into" and Perspectives

2007-05-29T10:37:00.000+10:00

I've finally started using Eclipse's "go into" functionality. Working with AONS 2, though not a giant project, was starting to get a little confusing; I was finding I was often moving between Java files, Spring configurations and various War resources (mainly JSP's). When you combine the weak checking going on between JSP's, SWF configurations and Java code, my short term memory proves to be quite inadequate. There is nothing worse then doing a build and realizing that I haven't changed a variable name in a SWF configuration file...

So, now working with my own deficiency taken into account, I have a perspective for:

Java
Run/Debug
Spring configurations
WAR resources

Using multiple screens to allow me to map (and check) between the various different screens is proving to be a much better option then trying to remember the latest iteration of variable name. It did take a little getting used to: previously I would only switch perspective when absolutely necessary, like when running/debugging. I've got two monitors too so I've configured Eclipse to open each perspective in a new application frame so I can drag them around. I've also made sure to name my task allocated perspectives (WAR Resource, Spring Resource etc).

All this will hopefully be unnecessary one day when all the different parts of a project are tied together with sensible logic checks; regular Spring configurations have been infinitely more usable with Spring IDE, and hopefully this continues with other Eclipse plugins.

As another option, maybe I should try training up my short term and long term memory... I hear memory mnemonics is good for that. One technique they say is to create a mental visualization of a place you've been, but every time I try visualise a mental "place" to put information I come up a little short. Maybe I should spend a little of my lunch time at the national library walking the floors to have a better mental picture....

JSR refactoring

2007-05-28T15:50:00.000+10:00

Maybe I've focused far too much on immediate problems as I've gone down my path as a Java developer and just trusted that out there, somewhere, someone was making good decisions for the Java community, leading us forward - tally ho and all that...

... and yet, looking towards the JCP for information on interface polymorphism support in JPA (JSR 220), I've come up a little sparse. The best thread I've found on the topic was offsite on the Hibernate forums (over here).

As the thread suggested, I emailed my "favourite" expert member of the JSR 220 to suggest support for interface polymorphism - and on the random wheel of potential support, I choose Linda DeMichiel (Michael Keith was a close second, but can't seem to find his email in my rolodex).

Something about this process seems a little light weight when compared with:

how much it affects a developer's daily life
just how many Java developers there are out there working with JPA in one form or another

So now truly seeing the JCP JSR process for the first time I can understand a bit more why JSR 306 is so important: we need to change this process in order to save it (or at least make it meaningful for developers). Transparency is key in most good development environments and the JCP JSR process has been left alone for too long. And yet, despite the momentum this JSR initially had, it appears it is falling behind schedule (snipped from here):

SR 306 passed JSR approval balloting unanimously. The current schedule indicates an early draft in December 2006 followed by a public draft in February 2007 . Final approval balloting is slated for May of 2007. InfoQ sat down with Heather VanCura, Manager of the JCP Program Office, and Onno Kluyt, Director of the JCP Program at Sun, and Chair of the JCP to discuss this new JSR and the changes it will be attempting to bring the JCP process.

So a public draft was due in February.... well it's almost June and still nothing. As the solution to many developers ills, it's running a little late.

JPA list ordering quirk

2007-05-25T13:27:00.000+10:00

Seems JPA has one of those quirks which just doesn't make Java sense. It seems that in the JPA specification, there is no preservation of list ordering, except on attributes based on the target object. The order of a list should not be tied to attributes on that list. Though it makes database sense, a lists state is independent of the contained objects in the Java world and thus it should be true also when using lists in JPA... I'm sure there were reasons for doing it this way, but it's kinda dumb.

JPA implementations usually create a many to many table for the join here anyway, so all it would have needed is an extra attribute identifying the order of the list.

Maybe I'm asking a lot, but I would have thought this would be taken as a given. As a side note, I am glad to find the Pro EJB 3: JPA book by Mike Keith and Merrick Schincariol covered this topic very well when I looked it up. Somehow Googling for "JPA list order" just didn't come up with anything very clear.

Jenkov Tree Tag and Spring Webflow

2007-05-16T08:44:00.000+10:00

I made a post over here about using trees in Spring Webflow (SWF); bascially I was asking if it was possible to do, and if so, what would be some good ways to go ahead. Since then I've implemented tree functionality within the AONS application to browse repository crawls. In order not to have to do everything myself, I integrated the Jenkov Tree Tag with SWF actions. The end result is quite workable and I'm rather happy with myself (don't worry, my ego usually deflates within 24 hours).

But, now at the end of the process, I can't help but offer some advice to those who'd also use the Jenkov Tree Tag (or the entire library).

Documentation, though quite large, is often incorrect
The tag library itself has some newbie errors (like for example giving every ".tld" file the same URI).
The approach of injecting a Tree Node Data Access Object (DAO) object into the tree is flawed; the Tree and DAO object must be serializable, so if it depends on other long lived beans the plan won't work. Instead, make the TreeNodeDAO a long lived bean, and whenever a SWF action is called upon the Tree, just replicate the functionality internal to the Tree in the action.
The documentation on the TreeUpdater() is just plain confusing.
The amount of hand crafting to create a good JSP tree structure can take quite a long time. You'll get there in the end, but it is a lot of trial and error - playing with tables, CSS, icons etc.

Still, kudos goes to the people over at Jenkov for putting this out there; Open Source libraries aren't always going to be fantastic and it takes a lot of guts and sweat to put a nice package together for usage - and any documentation is better then none.

Here's a picture of the tree utilised in the format summary page:

Some Java developers among you will notice the quite familiar icons from Eclipse: in theory Eclipse is open source so we can use them, but I'd hate to ruin any good feeling about AONS from the Open Source community so will take them out if asked (yet I'm almost equally loathe to work with 16x16 bitmaps in MS paint...).

[Update: After a little prodding, I've put up how I implemented the tree functionality over on the Spring Web Flow forum. I think my reluctance to detail how I did it was partly due to how big the explanation became (I had to turn it into three separate forum posts). The solution isn't perfect, in fact alternative solutions are welcome, but I do hope that it helps people out...feel free to post any questions (either here or on the blog)]

scheduling & transaction proxy wrappers

2007-05-12T10:42:00.000+10:00

Probably one of the most confusing parts of the last two applications I have written goes in the Scheduling area. I think I could be doing something better, but I'm just not sure how...

The area of complexity comes into play when Scheduling is mixed with transactions - not so much in that it's hard to enter a transaction, more in that it's confusing to update a "status" flag on a domain object in a new discrete transaction to indicate that the object is running. I've toyed with the idea that I could just have the status a memory based object, but that seems to smell even worse.

I guess further complexity is added by some databases locking for the duration of, what can be quite a long running, scheduled batch job. Most databases are okay with their database locks, but some are quite heavy handed in this, even when transactions are only read only. Luckily I'm currently testing against PostgreSQL and Oracle which, from past experience have shown to be quite smart in their locking processes so I probably won't have to cater for the unsaid bad database - whatever solution I talk about here, when the locking problem comes in too, things just get plain horrible.

So what's my ideal process for the transaction algorithm?
1.) Scheduled Job Starts, calls performSchedule on intended business manager
2.) Manager called, creating a new transaction
2.) New "tiny" transaction which commits quickly to indicate the domain object is running; there should also be a test in this small transaction to check that this domain object isn't already running.
3.) The batch main body of work goes on
4.) The batch main body of work is committed, we also indicate that the domain object is not running

Should a runtime exception be thrown, we should abort the main block of code, but in a catch block, we should also create another "tiny" transaction to indicate that the job is also no longer running and possibly do any logging/notification regarding the failed job.

Where this gets a little interesting is when transaction proxy wrapping mechanisms are thrown into the mix. Whilst not the worst code in the world by any means, we have to now employ a special mechanism to perform the two "tiny" transactions. There are a few ways to do this, some talked about here, but even those solutions do not handle changing the existing transaction demarcation around the proxy object. Since what we need to do is call a retrieve and and update which are normally PROPAGATION_REQUIRED in a very small transaction.

Say we are using business managers in our application as the target for the transaction proxies. Assuming that the business manager for the domain object is also running the scheduling (and here could be my problem), the we need to make the manager call another manager which is wrapped by a transaction proxy to make the normally PROPAGATION_REQUIRED retrieve and update methods wrapped inside a new transaction.

Trying to do this with Spring (at least with TransactionProxyFactoryBean methods), we get circular references which cause the context load to fail. Maybe this isn't the case anymore, but with Spring 2.0.2, this was still happening.

Another option here is to us something similar to the DAO template method where we use transaction callbacks to do a segment of work inside a transaction of our choosing. Hang on I hear you say, shouldn't the code be unaware of transactions since that was the whole point of transaction demarcation? Well yes, you're correct, but this isn't exactly a very normal situation - and in this case the situation requires a different approach since the code algorithm is implicitly aware of the transaction boundaries.

So in this solution, we create a transaction callback object, which would be a really basic interface with something like "TransactionCallback#execute()". This callback would then be passed to a TransactionManager... which would be wrapped by a transaction proxy. The TransactionManager would just have available methods like "executePropagationNew" and so forth.

This is by no means a perfect solution, but should be one of the best available considering.

Another possible solution would be to "break out" the scheduled methods from the relevant manager to a new manager. Still, even using this approach, the new manager would require the same implicit awareness of transaction boundaries and the resulting algorithm would be largely the same.

I hope that all made some sense - if I've confused you horribly and something isn't clear, feel free to ask questions and I'll try fix this up.

Serializable fumbling

2007-05-10T16:27:00.000+10:00

Just a small post today; whilst working with Spring Web Flow and Spring's JSP form:input tag, I was noticing one of my fields going blank. Convinced my code was somehow not mapping correctly, I looked everywhere about, trying to work out why it wasn't being kept.

I really should have noticed the significance sooner of the object reference changing between calls, even though in theory the object was instantiated only once. I think somewhere back in my mind I remembered a bit of a serializable issue once before, but unfortunately it didn't come up until after an hour of trying work out what was going wrong.

And in the end, the problem turned out to be that I hadn't defined my parent class (which held the disappearing variable) to be serializable, but had just done it on my child class. In essence, I'd missed a step in a valid serializable path.

Now, maybe I'm a bit cynical, but if a child class is defined serializable, isn't the parent implicitly serializable too? Obviously not, and thinking about it, I can sort of see why not since then someone could subclass something which was not designed to be serializable and whack on "implements serializable". I guess what I would have preferred happen is for the JVM to warn me about the potential issue here - maybe the Eclipse IDE should have too.

It's nice to find an area of knowledge which I didn't even realise I was weak in and correct that, but the road to understanding isn't always as fun as I'd like... whoa, very zen;P

format identification

2007-05-04T15:22:00.000+10:00

AONS version 2, like the first version will be utilising Droid and JHove for format identification. There is a bit of a catch 22 with regard to format identification tools and obsolescence/risk:

People have, as far as I am aware, mainly written identifiers for well used formats
Formats which are most "at risk" will be lesser used and probably won't have identifiers

So, there is a large chance that neither Droid or JHove will be able to identify important, yet lesser used, formats. If there is a large chance that important yet obscure files in a repository cannot be identified, we probably should ensure that the result of a repository scan includes the location, number and all known signifiers (maybe extensions?) of these unidentified files. A very useful application (maybe AONS 2.X or maybe AONS 3.X) would also allow a user to view singular unidentified files.

Now knowing that our repository crawl will give a metric for unidentified files, we should ensure that this metric is included when assessing the risk of obsolete documents within a repository - a repository with a high number of unidentified files should factor that in when viewing the obsolescence information.

Also - I'm a little worried about JHove/Droid's reliability in file format identification - looking at this post on the DSpace wiki makes me wonder whether extension based identification would be imperfect yet doable solution.

I think what I'll do is a bit of a combination approach:

Crawler gets a format resource as well as basic metadata about it (like location, full name, extension, mime type etc)
We have a list of format identifiers which will, for each, attempt to further identify the file (maybe determine version?) by using both the previous metadata and the actual file itself
Once all "identifiers" have run against the file, the final metadata should be at least as good as it was when we began the identification
With this potentially improved metadata in hand, we'll try and look for internal formats (those in the deployed AONS system) which match

Should a match be found between a repository format resource and an internal AONS format, we'll be able to link in the obsolescence information we have for that format. In the case where a link is not made, we will give the unidentified object, and any which have the same metadata the highest risk assessment possible.

I'm also going to try put in place relatively easy methods for repository owners to create basic mappings between extensions and mime types so that niche formats are easily identified.

Annotations - love to hate em

2007-05-01T12:39:00.000+10:00

When annotations work, they really do feel nice. But when they don't, oh boy, the hurt they inflict can be quite horrible.

I've been working with Compass and JPA annotations. I'm starting to get a bit better with Compass and was starting to do a bit more fine grained control over some of the fields which get stored but not indexed (via @Searchable(index=Index.NO)). Suddenly on a build I get the following error:

SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'registryDao' defined in ServletContext resource [/WEB-INF/spring/persistence-context.xml]: Cannot resolve reference to bean 'entityManagerFactory' while setting bean property 'entityManagerFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in ServletContext resource [/WEB-INF/spring/persistence-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalAccessError: class sun.reflect.GeneratedConstructorAccessor11 cannot access its superclass sun.reflect.ConstructorAccessorImpl
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in ServletContext resource [/WEB-INF/spring/persistence-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalAccessError: class sun.reflect.GeneratedConstructorAccessor11 cannot access its superclass sun.reflect.ConstructorAccessorImpl
Caused by: java.lang.IllegalAccessError: class sun.reflect.GeneratedConstructorAccessor11 cannot access its superclass sun.reflect.ConstructorAccessorImpl
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:588)
at sun.reflect.annotation.AnnotationParser.annotationForMap(AnnotationParser.java:239)
at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:229)
at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69)
at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52)
at java.lang.reflect.Field.declaredAnnotations(Field.java:1002)
at java.lang.reflect.Field.getAnnotation(Field.java:989)
at java.lang.reflect.AccessibleObject.isAnnotationPresent(AccessibleObject.java:175)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.MetadataHelper.isAnnotationPresent(MetadataHelper.java:653)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.MetadataHelper.isAnnotationPresent(MetadataHelper.java:668)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.accessors.MetadataAccessor.isAnnotationPresent(MetadataAccessor.java:377)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.accessors.ClassAccessor.isTransient(ClassAccessor.java:385)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.accessors.ClassAccessor.isValidPersistenceElement(ClassAccessor.java:408)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.accessors.ClassAccessor.isValidPersistenceField(ClassAccessor.java:417)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.accessors.ClassAccessor.processAccessorFields(ClassAccessor.java:525)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.accessors.ClassAccessor.processAccessors(ClassAccessor.java:552)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.accessors.ClassAccessor.process(ClassAccessor.java:483)
at oracle.toplink.essentials.internal.ejb.cmp3.metadata.MetadataProcessor.processAnnotations(MetadataProcessor.java:225)
at oracle.toplink.essentials.ejb.cmp3.persistence.PersistenceUnitProcessor.processORMetadata(PersistenceUnitProcessor.java:354)
at oracle.toplink.essentials.internal.ejb.cmp3.EntityManagerSetupImpl.predeploy(EntityManagerSetupImpl.java:584)
at oracle.toplink.essentials.ejb.cmp3.EntityManagerFactoryProvider.createContainerEntityManagerFactory(EntityManagerFactoryProvider.java:178)
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:204)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:242)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1118)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1085)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:429)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:250)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:141)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:247)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:161)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:245)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:124)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1019)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:809)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:425)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:250)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:141)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:247)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:161)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:273)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:346)
at org.springframework.web.context.support.AbstractRefreshableWebApplicationContext.refresh(AbstractRefreshableWebApplicationContext.java:156)
at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:246)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:184)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:49)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3763)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4211)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:809)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:698)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:472)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1122)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1021)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:718)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1013)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442)
at org.apache.catalina.core.StandardService.start(StandardService.java:450)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:709)
at org.apache.catalina.startup.Catalina.start(Catalina.java:551)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)

You'd think that some kind of sense would be made of that by a Java developer. But it's actually quite hard to begin digging for what was really going wrong:

What was the class which caused the error?
What was the annotation being processed?

Trying to look into or step through the java.reflect package yields a "source not found". Finally, where to we look for answers - I was able to guess that this error was in one of my annotations for Compass which was a bit of a stab in the dark but worked, but this error could as likely be related to Compass, JPA/Toplink or Spring.

So now having found the error - for some reason whenever I put on a "@SearchableProperty" on a member variable on one of my classes (it's just a plain old String variable...) I get this error. I've been able to workaround this error, but I can't see a way forward to identify which part of my technology stack is actually causing it - which makes the entire Java community all the poorer for it. If this is a common scenario in the community, are we headed for more fragile and confusing builds where debugging goes back to the opaque world it used to be.

In case anyone thinks of it, I'm already using the InstrumentationLoadTimeWeaver instead of a RelfectiveLoadTimeWeaver due to previous JPA/Toplink + Tomcat errors.

JPA collections of basic objects

2007-04-30T11:53:00.000+10:00

One small annoyance with JPA is that when using collections with generics of basic objects (String, Long etc) we have to explicitly create a wrapper entity to contain them. On the one hand, this approach is a more pure DBA conceptualisation since we have to explicitly think about our domain objects and can't just assume it's a Java list - and I'm sure this will make things simpler in JPA queries - yet it's a bit of a nuisance for every list in the domain model to be extruded into a separate entity.

It's a small annoyance, and if it makes the JPA queries easier to use then it's one we're probably going to have to live with;P

Compass in hindsight

2007-04-30T09:42:00.000+10:00

Okay, so now being a Compass veteran of at least a few days (har har, yeah okay, I'm still fumbling around in the dark), I think it's worth noting a few niggles I've found in it's usage:

Filtering results by class type is not easy within the index - I haven't yet tried it, but the issue is not covered in any of the supporting documents and this perspective is also reflected by this issue in Compass' JIRA installation. I'm pretty certain this is a common requirement. A workaround for this will be to maintain an index of all objects as a general search and also maintain indexes for each application domain object. Cumbersome, but should work (there are probably other methods of doing this by adding to the search string which I think I'll try out too).
Configuration error messages can be obtuse. Setting up the annotations necessary to get SearchableComponents will not be a simple job at first, and I found the error messages confusing. For one issue, which turned out to probably be a bug, I had to step through the code to see that setting root=false for a SearchableComponent was causing an error about mappings not being found.
Needs more examples on Collections - they do work, but the documentation does not give an actual example. Whenever I tried to get a SearchableComponent working which was a list and the component had no root, the list would fail - saying I should use generics when generics were in place (see the error message niggle).

Come to think of it, I think the (root=false) argument for a Searchable object probably just needs some rework in the Compass application. So, if you're going to use Compass as it stands at the moment, I'd avoid the (root=false) thingo altogether.

Oh yeah, one more thing which I'm not 100% understanding is the position that Compass takes to try and "rehydrate" the object from the index. My preferred option will be to use my existing DAO to retrieve the object based on the ID of the object found... My gut feeling is that this is the main use case for this piece of middleware, yet the documentation and examples don't really discuss this.

Hmmm, last thing, seriously. The GPS stuff in Compass is a little confusing - I'm using Spring and JPA; the Compass documentation is telling me I should be using the JPA GPS device, but I'm not exactly sure why - and if I do use this, how it changes any/all of the previous work I've done to integrate Compass. Moving on from the why, I'm not exactly sure how - though the written documentation is good, there is no example on integrating the JPA GPS device into a Spring application, just a few notes that it will not want a wrapped EntityManager.

So I'm still groping around in the dark a bit with Compass - but it is better then dealing with Lucene by itself... well at least that's what I'm still pretty certain of... hmmm

Compass for searching

2007-04-26T09:24:00.000+10:00

It seems that every time an application is created we always get to that point where we have to implement a nice search box to, well, find stuff. Many questions come up here:

how wide/specific should this search be (all domain objects or just one)
what kind of searching algorithm to use (literal matching or a relevance search algorithm similar to Google's Hilltop Algorithm)

Further, with REST as an exposed API into my application, how I'll translate a search on a REST CRUD object to an internal search will also be important.

I'm about to implement searching of external registry formats - the formats which a deployed instance of the application has downloaded from an external source. I think this is an area where I can definitely over think the problem to something which is too complicated for practical usage... so I'm going to try keep the solution relatively lean.

My goals here are to:
1.) Make cross field searching the default
2.) Keep the workload low
3.) Ensure the solution can be integrated well with Spring and my existing transaction borders

One nice little project which has come on my radar is Compass; it seems to do all of the above and more, plus has a really good set of user documentation.

So with Compass in hand, I'll see how I go.

One area which I'm a little confused as to how I'll move forward is in relation to, well, relations. I'd personally like to use the relations between objects to raise profiles/relevancy, but I haven't seen if this is possible with Compass.

Another area which I am unclear on is working the more traditional manager find methods with something like Compass. Assuming that under the hood my application uses Compass for indexing and searching, does each manager make calls into compass for searches (to something like a CompassManager) and then we also need to expose a global search function for all domain objects.

JPA Experience

2007-04-20T08:53:00.000+10:00

I've been using Toplink's JPA offering with HSQLDB as the databse platform. I'm really quite impressed with the offering as a whole:

Schema creation is a breeze
Annotations are pretty easy to use... I'm quickly getting hooked
Really feels like I'm able to quickly develop my domain model as I grow the backend

I did have a few niggles getting the application working in deployment though. Actually you could say there were a few very frustrating problems but were luckily picked up on the Spring forums - I'm glad it wasn't me to first find some of the issues with Tomcat and Toplink classloading. And I've got mixed impressions on the Weavers offered in Spring for getting the annotations working - they seem to work... but could use a bit of improvement.

The one big issue which I dislike is that I'm currently suckered with the InstrumentationLoadTimeWeaver which requires a -javaagent flag on my Tomcat startup. It's not the worst thing in the world, but I was hoping to get ReflectiveLoadTimeWeaver working but for some reason it wasn't picking up my annotations when combined with Tomcat/Toplink.

I've put the issue in the Spring forums, but so far no one has chimed in to either confirm they've had the issue or give me a pointer where I might be going wrong. I'm not crying sour grapes - help on the forum is only there if someone actually has the time to help and also the knowledge (and the issue is well written... maybe that's where I've gone wrong). Still, it feels horrible to still be suckered with that dang InstrumentationLoadTimeWeaver;)

Maybe I should try that Spring 2.0.4... I'm still on Spring 2.0.2

Manually Exposing Business Methods with REST

2007-04-16T09:00:00.000+10:00

After disqualifying the existing Java libraries to expose my business methods via REST in this blog, I started on my path of exposing my business methods via REST with my own design. So how hard was it to create a RESTful endpoint to my application? Well not that difficult really. In fact I'm quite amazed I got through it so quickly (Maybe I'm too cynical but nothing like this is usually problem free). I'll try and go through how my controller works in a logical order of operation. Feel free to borrow or take wholesale any of this for your own REST solution... in theory my project is open source so it should be fine (I'll put up details of the license when we've got it finalised, but code snippets should be okay for now). Also, there was some refactoring from a pure "inhouse Servlet" solution to a "Spring Model 2 Controller/View solution", but I'll leave that out as a private "lessons learned".

Tasks

Identify the target RESTful domain object type

The requirement here is that the portion of the URL after that identifying the target Servlet within the web archive should identify the domain object. So if we have a URL "http://localhost:8080/aons/rest/Registry" with:

http://localhost:8080 identifying the server IP address and TCP port
/aons identifying the web application
/rest identifying the servlet
/Registry identifying the target domain object

After refreshing my knowledge on all the little goodies in HttpServletRequest, I found the needed call in HttpServletRequest#getPathInfo(). We do have to remove the prefix forward slash, but apart from that, we've got a nice solid way of retrieving a target domain object type.

So the code becomes:

String pathInfo = request.getPathInfo();
if (pathInfo == null) {
throw new RuntimeException("Servlet path information is null so we are unable to determine the domain object type.");
}

domainObjectType = pathInfo.replaceFirst("/", "");
logger.info("Domain object name is: " + domainObjectType);

Easy as pie (not to be confused with PI.. ba-doom-tish, chortle chortle):)

Okay, no more (bad) puns, I promise

Identify the target method and override mechanism (for testing)

Identifying the target method in a RESTful application is quite important - soo many applications don't place much thought or checking in what methods they accept, and usually that's fine. Not here - now we're making the method type dictate what kind of operation we're performing on the given REST domain object. So, getting the request type parameter - again, easy as pie thanks to HttpServletRequest#getMethod(). The only real item for consideration here is that we probably want to override the method type so developers can hand craft URLs to invoke RESTful methods. We do this below:

String methodType = request.getMethod();
if (request.getParameter("methodType") != null) {
String overridingMethodType = request.getParameter("methodType");
logger.info("Overriding default method [" + methodType + "] with [" + overridingMethodType + "]");
methodType = overridingMethodType;
}

Should the parameter methodType be included, we'll choose a different option then that included with from the HttpServletRequest#getMethod() call. I'm pretty certain there are security and caching considerations if this style of overriding was used in production so it's probably worthwhile putting in place a testing flag to disable this mechanism. Still, it is probably worthwhile a discussion regarding the method override mechanism for doing more then the four CRUD options allow.

Create the mapping interface for a RESTful object to business managers

This task is probably where most of the variation in different REST solutions exists. For my solution, I wanted to favour simplicity over extensibility: My interface basically just gives the domain object type it maps, and the four crud methods to invoke depending on the methodType. So the initial interface is:

/**
* Transformation class which maps a REST method for a
* domain object to an underlying business object
*
* @author DLEVY
*/

public interface DomainHandler {

/**
* Retrieve the domain object type which this interface maps to
* @return the domain object name handled
*/
public String retrieveDomainObjectName();

/**
* Handle client calls for a HTTP GET request
*
* @param request servlet request
* @return object requested of this domain type
*/
public Object handleGet(HttpServletRequest request);

/**
* Handle client calls for a HTTP PUT
*
* @param request servlet request
* @return the identifier for the new object
*/
public Long handlePut(HttpServletRequest request);

/**
* Handle client calls for a HTTP POST call which is typically updating
*
* @param request servlet request
*
*/
public void handlePost(HttpServletRequest request);

/**
* Handle domain calls for HTTP DELETE call
*
* @param request servlet request
*/
public void handleDelete(HttpServletRequest request);

}

At this point it's probably worthwhile mentioning some of the other options available to me in a solution. I could have used a mechanism which utilised reflection and annototations to seek REST methods on business objects. Whilst the benefits of annotations are definitely real, I figured that for my situation it was probably overkill since the above solution works pretty easily, is simple and nice and easy to understand. I could have also used a generic handler call for all method types like handleCall(HttpServletRequest request, String methodType). That would have left a specific handler to interpret the call and decide whether the operation could be performed. Again, it didn't really seem to make that much sense to do this - whilst it would have added another layer of abstraction the benefits would have been minimal. So in the end I went with this, and it's been going pretty well so far.
In order to retrieve the handler in the AonsRestController, we use a map abstraction mechanism:

Retrieve the intended domain object type
Use the domain object type as a key in a map to the DomainHandler

This code which retrieves the appropriate DomainHandler in AonsRestController can be seen below:

if (singleDomainHandlerMap.containsKey(domainObjectType))
{
DomainHandler handler = singleDomainHandlerMap.get(domainObjectType);
}
else
{
String message = createAvailableDomainTypeString(domainObjectType);
throw new RuntimeException(message);
}

An example of an implementation GET method for the Registry handler in AONS is like so:

public Object handleGet(HttpServletRequest request) {
Long registryId = handlerUtil.retrieveParameterAsLong(request, "id", true);
return registryManager.retrieveRegistry(registryId);
}

On the DomainHandler we also put a method identifying which of the REST CRUD methods we implement, since not all domain objects will support the full stack:

/**
* Retrieve a list of methods we handle.
*
* The possible values returned here are:
* - GET
* - PUT
* - POST
* - DELETE
*
* @return list of possible values
*/
public List retrieveHandledMethods();

To call this class, we have the following logic in our controller. It's very "if/else", but since we're only dealing with four operations, it was something where the cost of putting in an abstraction layer out weighed the benefit of doing so:

if (!handler.retrieveHandledMethods().contains(methodType))
{
throw new InvalidMethodTypeException("Can not perform method [" + methodType + "] on [" + domainObjectType + "], available methods " + handler.retrieveHandledMethods());
}

// Okay, I could have used fancy reflection here
// but at the end of the day, the if else block
// really isn't too bad... and it's quite simple.
if (methodType.equals(RestMethodTypes.DELETE)) {
logger.info("We are performing a singular " + methodType + " operation on [" + domainObjectType + "]");
handler.handleDelete(request);
mv = new ModelAndView("NoReturnSuccessView", model);
} else if (methodType.equals(RestMethodTypes.POST)) {
logger.info("We are performing a singular " + methodType + " operation on [" + domainObjectType + "]");
handler.handlePost(request);
mv = new ModelAndView("NoReturnSuccessView", model);
} else if (methodType.equals(RestMethodTypes.PUT)) {
logger.info("We are performing a singular " + methodType + " operation on [" + domainObjectType + "]");
handler.handlePut(request);
mv = new ModelAndView("NoReturnSuccessView", model);
} else if (methodType.equals(RestMethodTypes.GET)) {
logger.info("We are performing a singular " + methodType + " operation on [" + domainObjectType + "]");
Object object = handler.handleGet(request);
model.put("object", object);
mv = new ModelAndView("SingularGetSuccessView", model);
}

Plural ("lists") as virtual domain object types

One "nice to have" identified was the ability to perform searching on domain objects. Since searching doesn't really fit into the four CRUD operations easily, I decided that I needed another mechanism to do this. I decided that when a person wants to do this in a RESTful world, they are really doing work on a virtual domain object. So for my code which concentrates on a Registry domain object, the plural Registries is the virtual domain object used for searching. The plural domain object only supports the GET call: no writes can be done at this level so PUT, POST and DELETE are all ignored. For convenience, I added the plural methods to the DomainHandler interface rather then requiring creation of actual DomainHandler implementations for these plural objects.

Below are the additional methods on DomainHandler:

/**
* Retrieve the plural version of this domain object for performing
* GET lists of this object. If there is no plural version of this
* domain object we just return null.
*
* @return plural object domain type for this handler
*/
public String retrievePluralDomainObjectName();

/**
* Handle client calls for a GET plural request which is typically a search
* @param request servlet request
* @return list of desired objects which may be empty
*/
public List handleGetPlural(HttpServletRequest request);

The actual AonsRestController uses this by having two key map objects, one for singular domain object types and one for the plurals. This was probably the more confusing part of the controller and I'll be looking to simplify it in future... but seems to work for now. The confusion is in working out in the controller if the domain object referenced is a singular or plural and then creating a pseudo CRUD method type GET_PLURAL. This pseudo CRUD method is internal only - the client still only passes a GET method with the plural domain object name.

Binding incoming parameters

This was one of those things which could have been rather difficult, but since I was able to tap into Spring Webflow's WebDataBinder, the job was simplified significantly. The nice thing is that even nested properties aren't that difficult to get working - though I did find it annoying that it doesn't automatically create nested objects with a default constructor or allow for one to register a factory to perform this operation. In my scenario I found this out when trying to submit a new Registry object with a nested Schedule object inside it. I found I had to create the Schedule object before performing the binding. For simpler structures, this isn't a big deal, but should input get relatively complex and be unknown at compile time it may pose an issues.

Here's an example usage of the WebDataBinder:

public void handlePost(HttpServletRequest request) {
Long registryId = handlerUtil.retrieveParameterAsLong(request, "id", true);
Registry registry = registryManager.retrieveRegistry(registryId);

WebDataBinder binder = new WebDataBinder(registry, "registry");

binder.bind(new MutablePropertyValues(request.getParameterMap()));

registryManager.updateRegistry(registry);

}

There was one issue with Spring's Webflow WebDataBinder: it would not automatically instantiate nested beans. So my Registry object has a Schedule object; this would have to be created by me before invoking the bind operation. Though I found a nice way to do this by checking for "schedule.*" parameters in the HttpServletRequest, it is something which should have been handled by the WebDataBinder. My attempts to fix/change the functionality in BeanWrapperImpl met with little success: The code changes really started to go beyond a nice simple refactor with all the private, protected methods I would have needed to change. So, in the end, my final code block looks like this:

public void handlePost(HttpServletRequest request) {

Long registryId = handlerUtil.retrieveParameterAsLong(request, "id", true);
Registry registry = registryManager.retrieveRegistry(registryId);

createScheduleAsRequired(request, registry);

WebDataBinder binder = new WebDataBinder(registry, "registry");
binder.bind(new MutablePropertyValues(request.getParameterMap()));

registryManager.updateRegistry(registry);
}

private void createScheduleAsRequired(HttpServletRequest request, Registry registry) {
Enumeration parameterNames = request.getParameterNames();
while (parameterNames.hasMoreElements()) {
String parameterName = (String) parameterNames.nextElement();
if (parameterName.startsWith("schedule")) {
registry.setSchedule(new Schedule());
break;
}
}
}

The solution isn't 100% perfect, but should do the job nicely for all I need

Error Handling

This wasn't actually as hard as it could have been, but was well worth doing. For error handling I want to ensure that any exceptions in the controller are caught and put inside a ModelAndView object. This is then returned normally. The view which renders this then checks if it has a renderer for that exception or any of the superclasses of the exception. If it doesn't it rethrows the exception and lets the Servlet container render it. If we do have a renderer for that type of exception or it's superclass, then we render it as meaningfully as possible. It's a relatively simple and powerful mechanism and should mean we always give something back to a REST client which can be parsed. At the moment I've configured the application with a general exception handler for all top level exceptions and a few specific sub classes of RuntimeException which should be handled (missing parameters and the like).

The only downside I can think of with doing this kind of thing is that by the time the View object has chosen to try render the exception, the contentType for the HttpServletResponse has been chosen. If we can't render the exception and rethrow it, then it's necessary for the contentType to be set again by the container. I'm not 100% certain if it is possible to change this more then once, but if the error is rethrown again and this has already been set, I'm not certain if the servlet can change this.

Where are the Java RESTful libraries?

2007-04-11T16:00:00.000+10:00

Where are the good Java RESTful libraries to help out with a typical application which already has business managers/facades and wants to expose them with REST? I came across these ones listed below but had to disqualify them:

XFire - Only talk of REST in their issue manager, but don't have it listed as an implemented feature. I would have liked to use XFire since I've enjoyed their library for SOAP endpoints - though documentation is as usual the projects weakness.

Restlet - Strongly pushing the idea that everything from the business tier to the DAO is uneeded. Doesn't have anything which enables me to easily expose a relatively standard business interface with mappings to the CRUD operations.

Gomba - Ack, run away, run away, for the beast has been unleashed. The mission statement for that project is that modifying library code past APIs is good so why not do it from the beginning. Whilst there may be some virtue in that approach for prototypes which have no possibility of life beyond a short project, it will pretty much ensure that any relativley standard library upgrades are painful. Reading this kind of project statement, I again wonder why a library designed to expose my application intends to give me any advice on how to structure my internals or my coding methodologies.

Apache CXF - I wasn't really sure what to make of this project - it looks to be akin to JBoss' SEAM in that it will help you build applications from the ground up. It might be great, but I'm not going to bother learning an entire new integration framework for only one piece of the pie.

So where does that leave me... well not in a very happy place. I'm quite loathe to have to do any creation since this task should be answered by a solution in the industry/community - if RESTful methodologies really are popular there should be a pre-existing solution (maybe when JSR311 matures). Then I started chewing around a few ideas on how this kind of thing might work, and the more I thought about it, the easier it got. So as you'll see in the next section, I didn't end up using an existing library, but baked my own which has proved to be about a day and a half worth of work and covers the essential areas in a REST service:

URL conventions and persistence

Easy hand testing vi URL crafting

CRUD operations

Plural "list" operations, for searching

Exception wrapping

Incoming parameter binding (thanks to reuse of WebDataBinder from the Spring Webflow project)

Creating Formats

2007-03-21T13:57:00.000+11:00

Just a quick post to work through some potential methods for creating formats in our system:

user driven
crawl driven

The more obvious case is user driven; as in the user creates a new format manually. The not so obvious answer is crawl driven, where the system would create empty formats in the case where a repository crawl found new formats. We will have to cater for both cases.

REST methods and contextual help

2007-03-21T12:42:00.000+11:00

Introduction

Yesterday we were asked by APSR to implement methods on the REST interfaces which would allow the end user to receive contextual help based on which page they were viewing. This issue, though not technically impossible or even difficult, raises the philosophical debate over what the REST/service interfaces intention purpose is.

Background Information

The AONS 2 application is an APSR funded application which is intended to work within the prototype federated architecture which will connect public registries together. Since initial discussions, it has been marked that AONS 2 will work not only within the federated architecture, but also in environments which are more traditional and closed. It is intended that APSR will create a remote GUI frontend, COSI (Collections Services and Infrastructures) for AONS 2 and other modules. This GUI frontend will handle conversational and session scope. The backend modules which AONS 2 is among will respond to calls from the APSR COSI frontend via REST services. On the COSI interface, it is intended that we have contextual help, depending on where the user is in the application (for example, step 3 of creating a new format).

See AONS 2 Wiki for more information.

Possible Solutions

We have a few possible solutions to this:

Expose help via contextual "page keys"
Do not expose any help methods

Solution 1 - Expose Help

This solution is not technically hard to do, but will be met with limited success. AONS 2 will be created with both the remote APSR COSI interface and also a local monolithic interface. Assuming that the same workflows are represented in each, it would not be difficult to create help for both. The only negative here is if the APSR COSI interface has different workflows then the core AONS 2 application. In this case, it would be wrong for the AONS 2 application to hold information about for workflows which it has no knowledge about. The important point to remember with this solution is that each of the user interface "steps" may not have a one-to-one mapping to a REST method. Going from best practices on web services, they should be relatively coarse grained and stateless (only request scope, not session or coversational). Therefore, it would only be possible to tie help to steps taken from AONS 2 monolithic interface.

Solution 2 - Do Not Expose Help

This is not as dire as it sounds. This solution simply acknowledges the idea that coarse grained REST methods should not be tied to user interface workflows. According to best practices, this is the ideal solution. If we are also developing user interfaces in parallel to those being developed by APSR, we should be able to give them a dump of any help we have developed an allow them to link it in to their own pages.

Conclusion

I would have to say that I prefer solution 2 since it keeps the REST interfaces responsibility only with performing business logic and keeps conversational scoped help in that tier. However, seeing as solution 1 would not be a large piece of work to implement if our screens are the same (AONS 2 monolithic UI and APSR COSI UI) then it should be seen as a viable solution - as long are we create the same workflows

Upstream Inconsistencies

2007-03-21T10:35:00.000+11:00

Introduction

Part of the function of AONS 2 is to utilise external registries of information which enables repository owners to make decisions regarding the obsolescence of a format. One of the inhibitors towards doing this is an inconsistent structure between different registries and what they offer. Further, a lack of a 100% consistent public unique identifier (PUID) means that we'll face issues even recognising that formats from different registries are one and the same. This post will discuss these two issues:

How we will handle linkages between external formats with no guaranteed unifying identifier
How we will handle merges between vastly different data structures

Background Information

Part of our work is to reference external sources (registries) of information to aid us in our judgment of obsolescence. It is assumed that in order to do this, a repository owner will utilise two types of information:

External information which is usually subjective in nature are requires human decision making to turn into a format suitable for obsolescence rules.
Internal information which is objective and based on external subjective information (above).

The first information is utilised as the source for the second type. When the first type changes, we will need to re-evaluate the second. This implies:

A clearly expressed link between internal objective rules and external subjective rules
A notification process when external subjective information changes which was used as the basis of internal objective information
Easy to use application "flows" which allow users to work with these changes and re-evaluate them in a logical manor

Mismatching Identifiers

The first big step is to link internal information objects to external subjective information. We have a few options here in how we'll do this:

Auto-magically: Take the user out of the equation and just automatically look for matches in external registries. This is a "weak link" approach, is quite simple, but may serve the needs of our users. The main issue here is that with the PUID's not being 100% consistent a link which should be present it missed. Also, some repositories may only give us a name or mime type as the initial basis to create a format. In this case, we will have to "help" the system identify what it is and make the linkage.
Explicitly defined: Make these linkages explicitly and manually enforced. A repository owner, on creating a new object (like an internal objective format) will have responsibility to search for external objects and link to them.
Combination approach: This approach uses a bit of a mix between the two approaches; first we auto-magically link to external subjective formats on object creation, but we also allow the user to manually change/affect these linkages.

Solution 1 - Auto-magical Linkage

Auto-magically would work, but exactly how is a bit unclear. An example algorithm would be:

User begins creation of new internal objective object (like a format)
User enters known fields (Name and/or PUID)
Auto-magic linking happens

This process would not be too difficult to implement and may be all we need initially.

Solution 2 - Explicitly Defined Linkage

Explicitly defined would also work, but would require a bit more oversight from the user:

User begins creation of new internal objective object (like a format)
User searches for "basis" external subjective objects
User accepts linkages

Solution 3 - Combined Linkage Approach

The combined approach is similar to the both steps, but it auto-magically searches for the linkages.

User begins creation of new internal objective object (like a format)
User presented with "potential" linkages
User accepts linkages
User free to look for further linkages

Conclusion

I think out of all of them, I like the combined approach the best - it fills in the fields as much as is possible but still gives the end user control of the whole process.

Merge Disparate Datasets

I think with the differences in offerings between various external registries we must be careful how we proceed with our merge strategy. We have to balance usefulness versus exponential application complexity.

Possible Solutions

We currently have the following solutions to this problem:

Attempt to auto-magically merge external subjective repositories into internal objective objects which can be used for obsolescence rules
Evaluate each registry on its merits and provide partial auto-magical merging into internal objective formats.
Do not attempt to turn external subjective information into internal objective information, but instead ensure that external information is close at hand when users are making internal objective decisions to aid decision making.

Solution 1 - Auto-Magically Merge

I have used this approach before where datasets were so large that it removed the possibility that a user would have time to manually review any fix performed. This solution is less then ideal since it almost never can capture the intelligence of an end user and exponentially increases the complexity of the application logic.

Solution 2 - Partial Merge

This approach could work quite well, adding characteristics of solution 1 only where it was easy to do. The only problem with this as a solution is that different registries may offer suggestions on different fields, resulting in an uneven result.

Solution 3 - No Merge

This solution is probably one of the best we can aim for; it is simple and embodies the qualities of our project in that we only want to attempt functionality with a good return on investment but also gives the user enough usefulness to perform intelligent obsolescence rules.

Conclusion

I think of all of them, I like solution 3 the best: it's achievable and workable.

AONS 2 Development Blog Created

2007-03-21T10:32:00.000+11:00

Okay, in order to help refine ideas and also as a place to put things of interest, I've decided to put up an AONS 2 Developer blog. I'll aim to put up issues and crossroads as we face them. People reading this blog should also have a look at the AONS 2 Wiki which details more about the project.