- 2075 Bhadra · 5 marks
Short note: Sqoop and Flume
Short answer
Sqoop transfers bulk structured data between relational databases and Hadoop using MapReduce, with import and export commands. Flume collects streaming log or event data from sources, buffers it in channels, and writes it to HDFS or HBase via sinks.
Write a short note on Sqoop and Flume.
Sqoop
Apache Sqoop ("SQL to Hadoop") is a tool for efficiently transferring bulk data between relational databases (MySQL, Oracle, PostgreSQL) and Hadoop (HDFS, Hive, HBase).
- Import: RDBMS table -> HDFS/Hive/HBase.
- Export: HDFS -> RDBMS table.
- Uses MapReduce internally: the transfer is split into parallel map tasks, which gives speed and fault tolerance.
- Works with a JDBC connection.
sqoop import --connect jdbc:mysql://host/shop \
--username user -P --table orders \
--target-dir /data/orders -m 4
Flume
Apache Flume is a distributed, reliable service for collecting, aggregating and moving large amounts of streaming log/event data (web server logs, social media feeds) into HDFS or HBase.
Architecture of an agent:
Source --> Channel --> Sink
(web log) (buffer) (HDFS)
- Source: receives events from the data generator.
- Channel: temporary buffer (memory or file) between source and sink.
- Sink: writes events to the destination (HDFS, HBase, another agent).
- Agents can be chained to form multi-hop flows with fan-in and fan-out.
Difference
| Point | Sqoop | Flume |
|---|---|---|
| Data type | Structured, from RDBMS | Streaming logs/events |
| Nature | Batch, on demand | Continuous |
| Direction | Import and export | Ingest only |
✓ Answer checked by Claude Opus 5.5 (high effort)