Pig Execution#

Execution Modes#

  • Local Mode

  • Tez Local Mode

  • Spark Local Mode

  • MapReduce Mode

  • Tez Mode

  • Spark Mode

/* local mode */
$ pig -x local ...
/* Tez local mode */
$ pig -x tez_local ...
/* Spark local mode */
$ pig -x spark_local ...

/* mapreduce mode */
$ pig ...
$ pig -x mapreduce ...

/* Tez mode */
$ pig -x tez ...

/* Spark mode */
$ pig -x spark ...

Pig Execution Mechanisms#

  • Interactive Mode (Grunt shell) − You can run Apache Pig in interactive mode using the Grunt shell. In this shell, you can enter the Pig Latin statements and get the output (using Dump operator).

  • Batch Mode (Script) − You can run Apache Pig in Batch mode by writing the Pig Latin script in a single file with .pig extension.

  • Embedded Mode (UDF) − Apache Pig provides the provision of defining our own functions (User Defined Functions) in programming languages such as Java, and using them in our script.

Invoking the Grunt Shell#

Local mode MapReduce mode
$ ./pig -x local $ ./pig -x mapreduce

Batch Mode#

Local mode MapReduce mode
$ pig -x local sample_script.pig $ pig -x mapreduce sample_script.pig
mkdir pig
cp /etc/passwd ~/pig

vi id.pig

/* id.pig */

A = load 'passwd' using PigStorage(':');  -- load the passwd file 
B = foreach A generate $0 as id;  -- extract the user IDs 
store B into 'id.out';  -- write the results to a file name id.out
pig -x local id.pig
cat id.out/part-m-00000


$ pig -help
vi $PIG_HOME/conf/pig.properties
tail -f -n 99 /tmp/pig-err.log