Short note: Pig and Hive

Apache Pig

Pig is a high-level data-flow platform for analysing large data sets in Hadoop, developed at Yahoo. Programs are written in Pig Latin, a scripting language, which the Pig engine compiles into MapReduce jobs. It is good for ETL and for programmers who want to avoid writing long Java MapReduce code. It handles structured and unstructured data.

A = LOAD 'students.txt' USING PigStorage(',')
    AS (name:chararray, college:chararray, marks:int);
B = GROUP A BY college;
C = FOREACH B GENERATE group, MAX(A.marks);
DUMP C;

Modes: local mode and MapReduce mode. Features: fewer lines of code, built-in operators (JOIN, GROUP, FILTER, ORDER), and user-defined functions.

Apache Hive

Hive is a data warehouse system on top of Hadoop, developed at Facebook. It provides a SQL-like language, HiveQL, for querying and managing data stored in HDFS. Queries are converted to MapReduce (or Tez/Spark) jobs. Table metadata is kept in the Metastore.

CREATE TABLE students (name STRING, college STRING, marks INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
SELECT college, MAX(marks) FROM students GROUP BY college;

It suits batch analysis and reporting by SQL users, but is not for real-time, row-level transactions.

Comparison