We use cookies to improve our website and your experience when using it. By using our services, you agree to our use of cookies. To find out more about the cookies we use and how to delete them, see our privacy policy


Big Data: The Competitive Advantage Is Pursued With Real-Time Update

Big Data can be better exploited if data can be updated in real time. Data Synchronization is the tool allowing to pursue a consistent competitive advantage

Big Data can be better exploited if data can be updated in real time. Data Synchronization is the tool allowing to pursue a consistent competitive advantage

The theme of Big Data is becoming increasingly topical and its interest is increasing for those companies who feel the need to analyze data in a more timely, accurate and fast way in order to correctly interpret the business.
Companies have realized that the ability to analyze a high amount of data and unstructured information can become a good competitive advantage. At an ICT level, however, the problem does not arise only with reference to the  infrastructure or the databases used, but also (and perhaps especially) with reference to how these data sources can be updated.
The competitive advantage stems not only when we are able to analyze large amounts of data, but, above all, if we are able to do it in real time. Moreover, in certain sectors, the timing of getting data is a key success variable, i.e. when the previous night data are already old today or, at least, when the way data are collected does not allow to follow certain trends in real time.
In this scenario, tools like ETL (Extract, Transform, Load), historically used to load data, are not very effective for who has the need to retrieve data in real time, since ETL technique poses several limitations in the phase of updating databases such as, for example, the large amount of data or the need to perform the refresh of tables.
The alternative of ETL is represented by data-synchronization tools , which are proving to be much more effective for updating a database.
Data synchronization is based on the technique of capturing the changes generated at source (i.e. from the tables of the information system) and distributing them, in a very controlled way, on the data analysis system. It’s easy to imagine how much more efficient is to capture a change and to bring it to the table you need rather than trigger a full data reload process (technique often used, i.e., to populate a data warehouse ).
This type of technology brings immediately three benefits, not negligible for applications working with Big Data:

Distribute only the needed information : whether we need to get real-time data, or to read consolidated information, the ability to capture and distribute only changes allows a great saving of time and of network / bandwidth.

Real-time information : only the synchronization technique allows to get the information at the same moment in which they are generated. In fact, the information capture engine, which works both with triggers and the database journal, triggers the process which deploys data into the target tables right when the change is generated.

Capture information wherever they are : data synchronization is an effective way to communicate with as many databases as possible to make all the necessary information available, without the need for specific knowledge of each type of database.

The data synchronization technique is assumed to capture the three actions that can take place at the record level (Insert, Update and Delete) and replicate them on the tables configured in the target system. This technique, however, does not force to replicate the update action immediately (although this is generally the most widely used mode). If needed, there are several other ways to do this, such as the append mode, which allows to convert any action taken (insert, update and delete) into actions queued on specific tables. This system allows to keep a history of updates, which is particularly useful to manage DWH issues, where the data historicization is often crucial.
Among the tools supporting data synchronization today on the market,  Duplo has been used for more than ten years in the management of ERP transactions and has been proving to be an effective tool particularly in the process of updating Big Data databases.
Duplo technical features, in fact, make it extremely flexible to support different ways of update, which is crucial in projects requiring updates to data analysis databases as well as when the need is for moving key business applications from a traditional database to a high performance one.