- 2075 Bhadra · 5 marks
Short note: Oozie
Short answer
Oozie is a workflow scheduler for Hadoop jobs, defining workflows as DAGs in XML with control and action nodes. It supports coordinator jobs for time or data triggers and bundles for multiple coordinators, and can rerun failed workflows.
Write a short note on Oozie.
Apache Oozie is a workflow scheduler system for managing Hadoop jobs. It lets users define a series of dependent jobs (MapReduce, Pig, Hive, Sqoop, shell, Java) as a single unit and run them in order, on a schedule or on data availability. It is a Java web application integrated with the Hadoop stack, and works with YARN.
Types of Oozie jobs
- Workflow job: a Directed Acyclic Graph (DAG) of actions defined in an XML file (
workflow.xml). It has control nodes (start, end, kill, decision, fork, join) and action nodes. - Coordinator job: triggers workflows at regular times (e.g. daily) or when input data becomes available.
- Bundle: a package of several coordinator jobs managed together.
start -> Sqoop import -> fork -> Hive job
\---> Pig job
join -> MapReduce -> end
(any failure -> kill)
Features
- Workflows are written in XML (hPDL); a job runs as a MapReduce launcher on the cluster.
- Supports fork/join for parallel actions and decision nodes for conditions.
- Sends email notifications and provides callbacks and a web UI/CLI for monitoring.
- Scalable and reliable: it can rerun failed workflows.
Use
For example, a daily pipeline: import sales data from MySQL with Sqoop, clean it with Pig, aggregate with Hive, and export results, all in one Oozie workflow.
✓ Answer checked by Claude Opus 5.5 (high effort)