XQuery

Chip a gem out of the monolith

Chip a gem out of the monolith 1000 1000 Nubisware

Leave all that can be spared behind. We travel light. Let’s hunt some orc. ― Aragorn, The Lord of the Rings

David’s success depended on surprise and accuracy. He knew he could not defeat Goliath on the giant’s terms, which is why he rejected Saul’s armor and with it the conventions of this form of combat. ― Lawrence Freedman, Strategy: A History

How an illuminated customer, carefully planned design decisions and a well chosen bunch of tools turned one of Nubisware’s most challenging commitments into a success story.

Late 2017 we’ve started to work at a very ambitious project for a large logistics company which had to face a complex refactoring of one of its core business applications.
The application had been built with an approach that at the time was considered traditional. A huge, web-based monolith rooted on top of a relational database.
Only very few technicians knowing what was going on under the hood and what were the design choices that led to particular solutions.
Up to when the supplier discontinued support, the application performed well enough. With the exception of a particular functional segment which had been causing headaches for years.
Our customer’s IT responsible was a very clever guy with a huge domain specific knowledge and a solid background in computer science. He was definitely tired of all the complaints raised by colleagues and customers so he asked the company’s CEO to invest in a refactoring of the indicted functionality. And that’s where Nubisware comes into play.

It must be said that, the functionality we were asked to refactor is a rather complex part of an already complex domain like logistics. The border between a rule and an exception is rather blurred. Thus, the work required a very careful analysis and planning which involved several meetings and a cyclic reworking of the project’s documentation.
Besides the inherent difficulty, the application also suffered from erroneous design choices that caused a general inflexibility making it very hard to correct weaknesses and limitations.
Relational persistence and the object-oriented data model proved to be major logical bottlenecks. We decided to go with a more data driven approach and helped the customer understand why a document oriented representation of the domain rules was better suited in this case.

From an architectural viewpoint the application was bulky, slow and to some extent unmanageable. We decided to rewrite the code as a set of cooperating microservices which are mostly stateless and thus perfectly suited to exploit inherent parallelism available on the customer’s very good infrastructure. Moreover, it enabled for a progressive migration towards the final solution involving several small and risk-limited integration steps.
Every major functionality (the algorithmic core, the data transformation services, the document repository, the graphical user interface) was split apart, wrapped with a REST API, rethought, rewritten, intensively tested and then put into staging. This was so surprisingly smooth that, to my feelings there has not even been an official staging-to-production phase. At some point have been told that users were using the new functionality with great satisfaction.
The serialization model for the documents was chosen to be XML because its formal solidity and the availability of validation and modeling tools granted the robustness that was needed for the backbone of the project. JSON was limited to the interaction between the User Agent, the GUI and some supporting services that provided integration to existing databases. The use cases were very heterogeneous but we wanted to keep the syntax (i. e. XSD) simple and manageable manually. Thus we decided to go for a very simple yet flexible XML structure with the possibility to enrich the semantics of elements through RDFa.
In order to manage the complexity and the continuously evolving specifications we decided to model all the application semantics (rules, constraints, relations, mappings, …) with an OWL Ontology designed with Protegé. Keeping most of the domain specific logic stored in an external, lightweight yet structured model allowed for late refactorings and enhancements to the application logic and proved to be also a great support for the implementation of the GUI.
All the code was written in XQuery 3.1 modules and BaseX was used as the runtime for the scripts and the REST APIs. The seven services were packaged into WAR archives (each less than 20MBs in size) and deployed inside Tomcat (7, 8, and 9).

In conclusion, the services have been running, nearly without intervention, for about 9 months by now. The customer is very satisfied about the huge improvements in flexibility and performance and, besides all the other metrics that we’ve taken (footprint, memory usage, throughput, …), this is the only actual one that counts at the end!

BaseX

Nubisware sponsors BaseX

Nubisware sponsors BaseX 1499 905 Nubisware

The reward for work well done, is the opportunity to do more. – Jonas Salk –

No person was ever honored for what he received. Honor has been the reward for what he gave. – Calvin Coolidge –

In order to explain in few words what BaseX is, I’d just cite the incipit of it’s home on the web which I warmly encourage you to have a look at.

BaseX is a robust, high-performance XML database engine and a highly compliant XQuery 3.1 processor with full support of the W3C Update and Full Text extensions. It serves as excellent framework for building complex data-intensive web applications. 
It comes with interactive user interfaces (desktop, web-based) that give you great insight into your data.

For us it has grown over the years to become an invaluable “swiss army knife” that we use pervasively to implement all sort of components for our customers’ micro-services, REST APIs, data-oriented applications and integrations. Not to speak of Nubisware’s own internal tooling and artifacts.

Great! At what cost comes all this power? Actually at no cost! Thanks to BaseX Gmbh, BaseX is usable for free with a very permissive opensource License.

For more than ten years, as employees of different companies, we’ve been very active BaseX users and we contributed to it with suggestions and code snippets, participating in the mailing list and even presenting our work at one BaseX users’ meetup in the context of XMLPrague (2015).

Today, joined into Nubisware, we are exited to announce that we are able to contribute back with a concrete sponsorship for the development of one awesome feature in the upcoming 9.x series that will even more boost our productivity and enhance the solutions we are developing for our most valuable partners and customers.

We are really looking forward to further collaborate with the wonderful staff at BaseX Gmbh!