Page History

...

### Experience Level: Intermediate / Advanced

## Abstract:

Resource Managers like Apache REEF (Sergiy Matusevych, Microsoft) - Resource managers like
Apache YARN and Mesos have emerged as a
critical layer in the cloud
computing system stack, but the developer
abstractions for leasing
cluster resources and instantiating
application logic are very
low-level. This flexibility comes at a high
cost in terms of developer effort, as each application must repeatedly
tackle the same challenges, like We present Apache REEF, a powerful yet simple framework
that helps developers of big data systems to retain fine-grained
control over the cloud resources and address common problems of
fault-tolerance, task scheduling and
coordination, and re-implement common mechanisms like caching and
caching,
interprocess communication, and bulk-data transfers.

We present Apache REEF, a development framework that provides a
control-plane for scheduling and coordinating task-level (data-plane)
work on cluster resources obtained from a Resource Manager. REEF
provides mechanisms that facilitate resource re-use for data caching,
and state management abstractions that greatly ease the development of
elastic data processing work-flows on cloud platforms that support a
Resource Manager service.

...

We will guide the
developers through a simple REEF application and dicsuss current state
of Apache REEF project and its place in the Hadoop ecosystem.

## Audience:

Apache REEF (Sergiy Matusevych, Microsoft) - This talk is for the
developers of big data systems. By guiding the audience through a

### Intended Audience: Big Data Systems Developers

By guiding the engineers through a simple Apache REEF application, we
give a low-level introduction to
big data systems, and show how Apache
REEF bridges the gap between
traditional big data systems like Apache
Spark, and low level API of
resource managers like Apache YARN or
Mesos. We demonstrate how Apache
REEF povides the basic control flow
for data-driven applications and
gives the developer a powerful
collection of general-purpose big data
tools, like bulk data
transfers, task management, and fault tolerance,
without imposing any
specific programming model. We also discuss the current state of the
Apache REEF project and its place in the Hadoop ecosystem.

## Linux/OSS gain:

Apache REEF (Sergiy Matusevych, Microsoft) - Originated at Microsoft
Cloud and Information Services Lab and
recently graduated as to a top
level Apache project, Apache REEF is yet
another evidence of Microsoft
commitment to support open
source. Battle tested in multiple
large-scale production systems,
Apache REEF fills an important niche
in the Hadoop ecosystem, and
opens Hadoop to the community of C#/.NET
developers.

## Speaker Bio:

Sergiy is a research engineer at Microsoft Cloud and Information
Services Lab, where he is building large scale distributed systems for
big data and machine learning. He is a committer to the Apache REEF
project. Prior to Microsoft, Sergiy worked as a data research engineer
at Yahoo! Research, and tried his hand in building machine learning
systems at several Silicon Valley startups. Sergiy is interested in
machine learning, data stream processing, and high performance
distributed systems. His previous speaking experience includes high
profile conferences like UAI,
MLDS, and HadoopSummit conferences.