Why do I need Data Integration?

In most cases an organisation grows from a small business into a larger business and along the line the data sources do the same. When you first started out, there might have been a single data base, but as Technology changed, and Development standards and people changed, more and more data sources were implemented on different technologies.

We have all heard that comment “But we can’t switch it off, there is valuable data in that database”, this is an issue many businesses faces today. Data Integration solves this problem, taking data from different data sources and bringing them into a single view for reporting and other operational activities.

Data Integration is also needed where the data sources and data storage destinations are different in technology and cannot communicate with each other. For example, a Data feed from a Kafka data queue brings the data into the environment, but you want to load this to a Microsoft SQL server data base. Or, Network data is received in high speed from a file-based network device and needs to be decoded and loaded to a Big data store.

There are other scenarios as well that we can discuss.

How would one go about this Data Integration?

In most cases the technologies that a business has in place can integrate from one technology to another, either with an add-on, or custom code that is developed by a developer.

There are also Data Integration platforms that can be used, these are high speed data processing and data transforming platforms that allow connections to multiple data sources as native connections. These Data Integration platforms are Clover ETL Data Integration, Microsoft SQL Server Integration Services, Ab Initio Data Integration, Informatica, Data Stage, Talend and many other opensource connections.

Based on the need of the customer DataSimplified can suggest a preferred technology based on what is available at the customer or an alternative based on functionality and available budget.

DataSimplified On Data Integration

Data Simplified have Expert skills in all the mentioned Integration layers, with specific focus on Clover ETL, Ab Initio and Microsoft SSIS.


What problems can I solve with a Data Integration solution?

Most businesses use a Data Integration layers to solve some of the following problems:

  • Slow data loading into Data Warehouse
  • Many times the “out of the box” connectors to load data into a data store is simple and slow, or there are multiple technologies involved in the process and native support for reading and loading data is not available.

  • Complex data processing in the Data Warehouse is slowing it down
  • So data processing and transformation is moved to the ETL layer (Extract Transform and Load) – Based on this approach the data is transformed outside of the data source and then loaded back in, as opposed to ELT that extracts the data, then loads it back into the Data source and then uses the data platform to transform the data.

  • Transforming data once for multiple data usages
  • This could also be a problem, where a single data stream enters the Business, but data needs to be prepared to be landed in multiple sources at the same time and on different technologies. This is typically not possible with an out of the box tool supplied with the technologies.

  • Enrichment of data from multiple data sources
  • is often a problem, since the data that needs to be used to enrich the source data might be in a different data source, or even worse it might not be available all the time., using complex ETL data integration platforms allows processing with in memory mapped lookup files that could retrieve data at a specific time and make available for enrichment to many other data sources when needed. So, no need to query enrichment data at run time.

  • Real time data processing can be very difficult to process
  • As most businesses are processing in a batch type environment, when a Real time feed is introduced the batch processing environment can often not integrate directly with the real time data source. With most Data Integration platforms this becomes possible and seamless.

With the technologies DataSimplified uses these problems and many other can easily be solved and implemented by using the correct Data Integration platform.
Since most businesses are only exposed to their own technologies or consultancies that put their own partner products forward, it can sometimes be very difficult to make an informed decision.
DataSimplified is technology agnostic and have a wide range of skills that can pick the correct tool for your environment, rather than being forced into a fixed list supplied by a vendor.

Contact us if you want to look at the perfect Data Integration platform to solve your data needs.