Trakti Blog

Deals made easy!

It’s time to abandon the myth of normalised data storage

by trakti on May 26, 2015 Comments Off on It’s time to abandon the myth of normalised data storage

Data modelling is till a fine art with severe impacts in programming as well as in persistency. Long discussions are necessary to decide the better balance between complexity, ease of use, comprehensibility, ORM mapping and reporting need. It’s a bad habit to first implement and then to let the evolution dictate how the schema evolve; impacts can be tough. It’s fine if we add attributes during sprints, also adding relations is fine. I’m in favour of iterative and incremental approach with short cycle, nevertheless a strategical decision is to undertaken before starting the project in relation to the chosen persistency strategy.

Data doubtsIf we are going NOSQL a different modelling approach is required. The classical ER approach were we model relations based on the dependencies of real life concepts have to be abandoned. In the DBA tables of law it was written: believe in normalised tables, your only God.! Or maybe not?

For example, in our case in Trakti, a negotiation is a process that proceeds by sending offers across parties, and where messages are exchanged across them. In a relations DB it’d be natural to model this scenario as a table of negotiations in relation 1:m with the table of offers. In the same way we’d define a relation with the table of messages.

On the other hand in a NOSQL DB we have to consider data structures that reflect how data is presented to the user, how data in general is manager on the UI and not how the concepts are related. We have to prefer grabbing data using less relations as possible. In case of presenting the details of our negotiation we do not want to navigate relations to fetch records in other tables, but rather a single read operation from a single table (a document in term of NOSQL). This approach recalls me the Value Object pattern of the pre-NOSQL times where we had to constructs this object in the application, in order to fulfil the need of the UI. As such we did not have to manually map the objects to the widgets The data was changing structure while moving across the tiers. NOSQL is more pragmatic: it allows us to bypass this complexity. The danger here is to always have a too big object, but we can work this out with a lazy load feature.

Nevertheless complexity can emerge when implementing reporting, not mentioning BI. In this case we have to look inside the documents, extracting the data needed for the BI, and in absence of relations we have to solve the problem in code: a rally pain for developers that badly affects productivity. Here we have to consider to be ready for adopting an ETL solution early in stage in the development.

NOSQL is fine, but we haver to be agile in revisiting what we have learned in the past, we have to admit for example that also for OLTP persistency we have have non normalised tables.

traktiIt’s time to abandon the myth of normalised data storage