Hybrid data in big data warehouse

Hybrid data is the mixture of structured, semi-structured and unstructured data (from internal and external, batch and streaming sources) that is stored and analysed together in a big data warehouse. A traditional data warehouse holds only structured, cleaned, schema-defined data. A big data warehouse also keeps data in its raw or varied form, so its data is hybrid.

Types of data found in a big data warehouse

Hybrid also refers to mixed origin and mode: data from enterprise systems (ERP, CRM), external sources (social media, sensors, web) and both historical (batch) and real-time (stream) data.

Why it is called hybrid

• The warehouse combines relational tables with files in HDFS/NoSQL stores.
• It uses both schema-on-write (cleaned structured data) and schema-on-read (raw data interpreted when queried).
• It integrates batch (Hadoop) with real-time (Storm/Spark Streaming) processing.

 ERP/CRM (structured) --\
 Logs, JSON (semi) -------> Big data warehouse --> Reports,
 Tweets, images (unstr.) -/  (HDFS + NoSQL + DW)    analytics

Examples

1. Retail company: sales transactions (structured) joined with website click logs (semi-structured) and customer reviews and photos (unstructured) to understand buying behaviour.
2. Telecom: call detail records (structured), network logs (semi-structured) and customer-care call recordings (unstructured) for churn analysis.
3. Hospital: patient records (structured), doctor notes and X-ray images (unstructured), and sensor readings (semi-structured) for diagnosis support.
4. Bank: transactions (structured), e-mail and chat transcripts (unstructured) for fraud detection.

Benefits

A complete 360-degree view of the customer, richer analytics than structured data alone, and flexibility in adding new data types.

Challenges

Integration of formats, data quality, metadata management and security.

Traditional versus big data (hybrid) warehouse

How hybrid data is handled

1. Ingestion: Sqoop moves relational data into HDFS, Flume or Kafka collect logs and streams.
2. Storage: raw files stay in HDFS (data lake), refined structured data goes to Hive tables or the existing DW.
3. Processing: MapReduce or Spark transform and join the different forms; unstructured text is parsed into features.
4. Access: analysts query through Hive or SQL-on-Hadoop and visual tools, so that structured and unstructured results are viewed together.
5. Governance: metadata catalogue, data lineage and access control span all data types.