---
title: "Data integration and its challenges"
subject: "Data Mining"
chapter: 2
key: "semester_7/data_mining:data-integration"
source: https://www.bctengineer.com/q/semester_7/data_mining/data-integration
site: BCT Engineering
---

# Data integration and its challenges

Data Mining · Chapter 2: Data Pre-processing

- Asked in: year not given
- Video: [L26: Data Integration | Issues in Data Integration | Data warehouse and Data Mining Lectures Hindi](https://www.youtube.com/watch?v=UKUq7hZdZUw) — Easy Engineering Classes
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_7/data_mining/data-integration
- Whole chapter: https://www.bctengineer.com/md/semester_7/data_mining/chapter-2

## Question

What is data integration? What are the challenges of Data Integration? How can they be handled?

## Short answer

Data integration merges data from multiple sources into a unified view. Challenges include schema conflicts, redundancy, value conflicts, and duplicates. They are handled by metadata matching, correlation analysis, standardizing units, and record linkage.

## Answer

**Data integration** is the merging of data from multiple sources (databases, data cubes, flat files, web) into a single coherent store such as a data warehouse, so that analysis can be done on a unified view.

#### Challenges and how to handle them
| Challenge | Explanation | Handling |
| --- | --- | --- |
| Schema integration / entity identification | Same entity has different names, e.g. `cust_id` and `customer_no` | Use metadata (data dictionary) to match attributes; match keys and structure |
| Redundancy | An attribute can be derived from another (age and date of birth), or the same attribute repeats | Correlation analysis: χ² test for nominal, correlation coefficient r for numeric data; remove one of a correlated pair; detect duplicate tuples |
| Data value conflicts | Same attribute has different representations or units (kg and lb, Rs and USD, different date formats) | Convert to a common standard, scale and unit before merging |
| Heterogeneous data models | Relational, XML, text, different DBMS | Use wrappers/mediators or ETL tools to convert to a common model |
| Duplicates and inconsistency | Same record stored twice with different spellings | Record linkage / de-duplication, normalisation |
| Semantic differences | A "customer" in one system differs from another | Agree on common definitions, ontologies |
| Volume and time | Data size, update frequency | ETL scheduling, incremental loading |

Careful integration reduces redundancies and inconsistencies and improves the speed and quality of the later mining.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is data integration?
   A. Splitting one table into many
   B. Deleting duplicate attributes only
   C. Merging data from multiple sources into one coherent store
   D. Sampling data
   Answer: C. It gives a unified view, e.g. in a data warehouse. (Key point: Definition)

2. How is attribute redundancy detected?
   A. By sorting tuples
   B. By binning
   C. By checking file sizes
   D. Correlation analysis: chi-square for nominal, correlation coefficient r for numeric
   Answer: D. A highly correlated attribute can be removed. (Key point: Redundancy detection)

3. How are conflicts like kg vs lb handled?
   A. Convert to a common standard, scale and unit before merging
   B. Ignore them
   C. Delete the attribute
   D. Average the numbers
   Answer: A. Representations are standardised first. (Key point: Value conflicts)

4. How is cust_id vs customer_no matched?
   A. By random guessing
   B. Using metadata such as a data dictionary
   C. By dropping both attributes
   D. By clustering the tuples
   Answer: B. Metadata helps to match attributes referring to the same entity. (Key point: Entity identification)

## Related in Data Mining

- [Anomaly detection challenges and schemes](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aanomaly-challenges-schemes) — Data Mining, chapter 6
- [Web mining structure and challenges](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aweb-mining-structure-challenges) — Data Mining, chapter 7
- [Data warehouse, data mart and need](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Awarehouse-datamart-need) — Data Mining, chapter 1

## Related in other subjects

- [Big data trends, challenges, characteristics](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Abig-data-trends-challenges) — Big Data Technologies, chapter 1
- [Handling voluminous data](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Avoluminous-data-techniques-challenges) — Information System, chapter 8
- [Simple curve data, roads intersecting at 150°](https://www.bctengineer.com/question.md?key=semester_4%2Fsurveying_ii%3Asimple-curve-two-roads-150-4deg) — Surveying II, chapter 6