In most cases an organisation grows from a small business into a larger business and along the line the data
sources do the same. When you first started out, there might have been a single data base, but as Technology
changed, and Development standards and people changed, more and more data sources were implemented on different
We have all heard that comment “But we can’t switch it off, there is valuable data in that database”, this is an issue many businesses faces today. Data Integration solves this problem, taking data from different data sources and bringing them into a single view for reporting and other operational activities.
Data Integration is also needed where the data sources and data storage destinations are different in technology and cannot communicate with each other. For example, a Data feed from a Kafka data queue brings the data into the environment, but you want to load this to a Microsoft SQL server data base. Or, Network data is received in high speed from a file-based network device and needs to be decoded and loaded to a Big data store.
There are other scenarios as well that we can discuss.
In most cases the technologies that a business has in place can integrate from one technology to another, either
with an add-on, or custom code that is developed by a developer.
There are also Data Integration platforms that can be used, these are high speed data processing and data transforming platforms that allow connections to multiple data sources as native connections. These Data Integration platforms are Clover ETL Data Integration, Microsoft SQL Server Integration Services, Ab Initio Data Integration, Informatica, Data Stage, Talend and many other opensource connections.
Based on the need of the customer DataSimplified can suggest a preferred technology based on what is available at the customer or an alternative based on functionality and available budget.
Data Simplified have Expert skills in all the mentioned Integration layers, with specific focus on Clover ETL, Ab Initio and Microsoft SSIS.
Many times the “out of the box” connectors to load data into a data store is simple and slow, or there are multiple technologies involved in the process and native support for reading and loading data is not available.
So data processing and transformation is moved to the ETL layer (Extract Transform and Load) – Based on this approach the data is transformed outside of the data source and then loaded back in, as opposed to ELT that extracts the data, then loads it back into the Data source and then uses the data platform to transform the data.
This could also be a problem, where a single data stream enters the Business, but data needs to be prepared to be landed in multiple sources at the same time and on different technologies. This is typically not possible with an out of the box tool supplied with the technologies.
is often a problem, since the data that needs to be used to enrich the source data might be in a different data source, or even worse it might not be available all the time., using complex ETL data integration platforms allows processing with in memory mapped lookup files that could retrieve data at a specific time and make available for enrichment to many other data sources when needed. So, no need to query enrichment data at run time.
As most businesses are processing in a batch type environment, when a Real time feed is introduced the batch processing environment can often not integrate directly with the real time data source. With most Data Integration platforms this becomes possible and seamless.