Skip to main content

Big data and its processing

Short answer

Big data is datasets too large, fast, and varied for traditional tools, characterized by volume, velocity, variety. Processing involves ingestion via Flume/Kafka, distributed storage in HDFS/NoSQL, batch processing with MapReduce or stream processing with Spark, then analysis and visualization.

Explain Big Data. How is it processed?

Big Data

Big data means datasets so large, fast-growing and varied that traditional database tools cannot store, manage or process them efficiently. They come from social media, sensors, e-commerce, logs and mobile devices.

Characteristics (the Vs)

  • Volume: terabytes to petabytes.
  • Velocity: speed at which data is generated and must be processed.
  • Variety: structured, semi-structured and unstructured data (text, images, video).
  • Veracity: uncertainty and quality of data.
  • Value: useful insight that can be extracted.

How big data is processed

  1. Data collection / ingestion: from sources using tools such as Flume, Kafka, Sqoop.
  2. Storage: distributed systems such as HDFS, NoSQL databases and cloud storage.
  3. Processing:
    • Batch processing: Hadoop MapReduce works on large stored datasets in parallel (map, shuffle, reduce).
    • Stream processing: Storm, Spark Streaming, Flink handle data in real time.
    • In-memory processing: Spark for fast iterative analysis.
  4. Analysis: querying (Hive, Pig), data mining, machine learning, text and graph analytics.
  5. Visualisation and reporting: dashboards and charts to support decisions.
 Sources -> Ingest -> Store (HDFS/NoSQL)
        -> Process (MapReduce/Spark)
        -> Analyse -> Visualise

✓ Answer checked by Claude Opus 5.5 (high effort)

Related questions

All Information System chapter 8 questions →