1. bigfootproject
  2. Untitled project
  3. Hadoop 0.21 with suspension

Overview

HTTPS SSH

Preemption for Hadoop

We deliver a patched version of Hadoop 0.21 which enables efficient task-level preemption.

The characteristics of this software are summarized in Deliverable 3.2 of the BigFoot project (cf. http://bigfootproject.eu/documents.html)

Compiling and Installing

This patched version of Hadoop is compiled and installed in the same way as upstream Hadoop version (cf. http://wiki.apache.org/ hadoop/GettingStartedWithHadoop). The source code is included in this repository.

Using Suspension

This patched version of Hadoop exposes new options to the command line. To suspend a task, it is sufficient to call the command hadoop job -suspend-task <taskid>

Tasks can be resumed via the command hadoop job -resume-task <taskid>

Schedulers can exploit the same functionality via the suspendTaskAttempt and resumeTaskAttempt methods present in the TaskInProgress class.