• DQaaS

    Data Quality as a Service

    Bottom Line impacted by your Data Quality?

Who needs data quality

A very simple answer to that question if everyone. From the small shop on the corner, to the medium sized Accounting firm, to the large corporate. Everyone needs to worry about their data!

Data quality is not something most companies worry about much, since they have what they think is manageable data, so a person can maintain it, and it is their job to ensure it is accurate.

As that company grows, they keep in believing that it is being maintained and the person they have is doing their job. Before they know it, the data volumes have increase and that person it taking strain ensuring that the data is maintained on a daily basis. This could be that one large customer that you bring on board as you grow your business, that one that is going to make you grow, the one we are all looking for. Your excited about the growth, and the last thing on your mind is that one person capturing the data and ensuring it is accurate. You soon find yourself adding more and more staff to the data capturing team to ensure that the data is updated and maintained. By the time you stop and look around, you have a team of people being paid to ensure that the main thing making your business possible is safe, and correct.

Unfortunately, there will be a time where people cannot maintain data any more, it needs to be done by a system to double check what has been input is correct. That one safeguard to remove the risk from your business to ensure you have one less thing to worry about. The truth is simply that smaller companies, and your SME, are staring to gather and store data at an incredible rate, and now I am not even talking about the corporate giants that generate and consume data in a scale we sometimes cannot imagine. The business world is becoming driven by data, if we like to hear it or not. And if you do not need a data quality solution now, you will need it very soon.

Questions you should ask yourself

Maybe the data was late? How do you currently manage your data suppliers, do you know how many times they send bad data, how many times the data arrives later than expected? Can you report historically how many times a specific data provider did not follow the data policy you supplied? Can you report to see what data type is always an issue for most partners? The solution needs to be able to answer all these questions.

What more should it do?

Taking it a bit further, the solution needs to assist you in managing your data providers, if bad data is received it needs to inform them of the issue and allow them to repair the data and resubmit. The solution needs to gracefully handle this resubmitted data and provide the corrected data to your internal systems.

Lastly the solution needs to store all this information, making it possible to report on historical data changes, and even load it to a Staging area for your Enterprise data warehouse or other internal systems. A good data quality solution will allow the business to use the data that is stored for many more applications than they previously thought possible, since it will form a holistic view of all data entering into the business over all time.

Can data quality be affordable?

Well we all know that small companies are fighting the cash-flow problem every day, they do not have the money to spend on buying software that manages this for them. As for SMEs they have strict budget constraints and will not get approval for a new investment for data quality that management thinks can be done by a data capture, or some expensive Dev-Ops resources.

With Multi-National corporations they will likely invest in a large-scale data quality program, to ensure they are on track, and based on estimates it looks like big numbers. It is estimated that the industry will spent $14.7 million per year on correcting data quality issues, and that the average corporation has 18 full time employees to just focus on Data quality.

So, the question needs to be asked, how can I get Data Quality that is right for my business, be it small, medium or large. You have three options available to you, buy a on premise Data Quality solution, develop a in house solution with your own Development teams to achieve data quality and lastly get a Data Quality as a Service solution.

On Premise Data Quality Solution

With a on premise Data Quality solution you will need to invest money in the following:

  • Once off Licencing cost (even maybe 3rd party licences to have the solution run in your environment)
  • Once off Cost of resources - You will need to setup a team to work with the supplier to install and configure it for your data feeds
  • Once off Installation fees or as they call it Implementation fees and Configuration
  • Yearly maintenance fees (Between 15% and 22% of the solution licencing cost
  • (Maybe) a Once off Hardware infrastructure purchase (This will also have an annual maintenance fee)
  • Setup a support team including IT personnel to maintain the solution and infrastructure
  • New versions of the software will be made available to stay up to date, but will required an upgrade strategy by your IT teams

In house Data Quality Solution

If you were thinking of going the route of implementing an in-house developed Data Quality solution the sky is the limit, you can spend as much as you like, or rather have available the following needs to be considered in such an approach:

  • A team of Data experts in the company will need to get together and share knowledge
  • All your data sources (internal and external) will need to be meticulously documented to ensure you know what you want to check for Data Quality
  • A project with budgets will need to be registered with your project office and resources from the development teams will need to be allocated
  • Third party licences will need to be procured to spin up instances for Development environments, data bases, and other licences that are required for development (if not available)
  • Hardware infrastructure will need to be provisioned for these environments
  • Development can start on the project and based on complexity and total data sources this could range from 6 months to 18 months of development cycles
  • IT teams are setup to monitor and support the infrastructure and solution
  • Once the first release is available business will finally realise what they have been missing and start requested changes and additions to the system to better get their requirements in place. (This always happens, so never fool yourself to think it will not make ready for an extended budget after phase 1 is completed.)

If you are a smaller company and do not have the development resources or the skills to implement it yourself, or you do not have the budgets required for buying a new product, you are left with Data Quality as a Service.

Data Quality as a Service

With Data Quality as a Service you can benefit in the following way:

  • No third-party licences for software or tools
  • No Development costs
  • Hosted solution is provided to you, so no technical IT skills is required to maintain the infrastructure or the solution
  • Disaster Recovery is inherent in a hosted environment, so no need to spend money on this (except on your side at the business for sending and receiving the data)
  • A Monthly OPEX payment is made instead of a single large CAPEX investment
  • This OPEX expense will be based on a monthly fixed fee and a data volume-based fee for processing
  • There is sometimes a free trial or POC period available to test the offering
  • Because a hosted solution will be shared by more than one customer this reduces the cost of the operational costs, allowing all customers to benefit from the cost saving
  • There will be an initial setup fee to configure your environment, but after that, there will be a self-service interface to maintain the setup and config, allowing you to add additional data sources and files
  • All new features being added to the DQaaS solution in the hosted solution will be available to all users with no upgrade strategy being required
  • You will still need your Data sources documented to ensure that the DQaaS can be setup on your behalf

Final Word

As a business owner, CTO, CEO or CDO you need to decide what best fits your requirements, based on the insight I have provided the choice should now be easier to make. There are multiple solutions available in the market, and many consultancy firms that can assist in making sure you reduce your data quality risk. Keep in mind the new GDPR requirements, as good data quality and data governance processes are key and need to be reported on. It is estimated by 2020 that more than 80% of organisations will have a newly appointed CDO (Chief Data Officer) and they will be solely responsible for the data governance of the organisation. We as business owners need to understand that the ownership of data is becoming the biggest focus of the international market and will keep on increasing over the next 5 years, we need to be ready and stay ahead of the requirements, once we fall behind it is extremely difficult to play catch up.