Data Leaders members can read the full summary of the peer exchange and connect with peers via the Data Leaders Hub.

In a recent Data Leaders peer exchange, data practitioners came together to share how they are deploying artificial intelligence to improve the efficiency of identifying and rectifying data quality issues. Here are five examples of how AI is helping them to enhance efficiency, accuracy, and consistency in data management while reducing manual efforts and improving overall data quality.

1. Data Integration & Entity Resolution

In this use case, machine learning algorithms are employed to match and merge customer data from various sources, creating a “golden record” that accurately reflects customer information. This approach includes deduplication, reconciling discrepancies, and resolving entities across datasets. Continuous training of these AI models, particularly for complex cases like matching similar international names, is crucial. A human-in-the-loop process ensures the algorithms are retrained when discrepancies arise, improving accuracy over time.

2. Anomaly Detection In ESG Data

To manage the complexities of ESG (Environmental, Social, and Governance) data, which is often unstructured and inconsistent, AI utilises fuzzy matching to identify and match companies while assigning probability scores to assess match accuracy. High-probability matches are processed automatically, while lower-confidence matches undergo manual review. One data practitioner also shared how AI models can be used for anomaly and outlier detection, enhancing the overall quality and consistency of ESG reporting.

3. Synthetic Data Generation

Applying AI to create synthetic data, especially in low-volume datasets, for example ESG data, better supports predictions and experiments.

4. Standardising Data Categorisation

Initial data catalogues are often filled with inconsistent acronyms and descriptions, leading to confusion and a lack of coherence. To address this data quality problem, AI, particularly large language models (LLMs), is used to standardise data categorisation. This use case focuses on cleaning up inconsistencies in existing data labels, helping standardise how data is described. By applying AI models to the catalogue, data leaders can reduce these variations and create more standardised terms and naming conventions, improving coherence and strengthening data governance.

5. Data Ingestion Issue Detection

Data ingestion issues can consume up to 20% of data engineers’ time. AI solutions are being explored to detect and address those issues proactively. By automating error detection to reduce manual intervention, improve efficiency and free up resources as data grows.

Ensure you get stories like this and many more interesting insights from data and analytics leaders like you – directly to your inbox – by signing up to our newsletter. Would like to become a member? Get in touch!