Page History

CFP is here: http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp

### Submission type: Presentation

...

### Experience Level: Intermediate / Advanced

## Abstract:

Resource Managers Apache REEF (Sergiy Matusevych, Microsoft) - Resource managers like Apache YARN and Mesos have emerged as a
critical layer in the cloud computing system stack, but the developer
abstractions for leasing cluster resources and instantiating
application logic are very low-level. This flexibility comes at a high
cost in terms of developer effort, as each application must repeatedly
tackle the same challenges, like We present Apache REEF, a powerful yet simple framework that helps developers of big data systems to retain fine-grained control over the cloud resources and address common problems of fault-tolerance, task scheduling and
coordination, and re-implement common mechanisms like caching and
caching, interprocess communication, and bulk-data transfers.

We present Apache REEF, a development framework that provides a
control-plane for scheduling and coordinating task-level (data-plane)
work on cluster resources obtained from a Resource Manager. REEF
provides mechanisms that facilitate resource re-use for data caching,
and state management abstractions that greatly ease the development of
elastic data processing work-flows on cloud platforms that support a
Resource Manager service.

...

We will guide the developers through a simple REEF application and discuss current state of Apache REEF project and its place in the Hadoop ecosystem.

## Audience:

Apache REEF (Sergiy Matusevych, Microsoft) - This talk is for the developers of big data systems. By guiding the audience

### Intended Audience: Big Data Systems Developers

By guiding the engineers through a simple Apache REEF application, we
give a low-level introduction to big data systems, and show how Apache
REEF bridges the gap between traditional big data systems like Apache
Spark, and low level API of resource managers like Apache YARN or
Mesos. We demonstrate how Apache REEF povides provides the basic control flow
for data-driven applications and gives the developer a powerful
collection of general-purpose big data tools, like bulk data
transfers, task management, and fault tolerance, without imposing any
specific programming model. We also discuss the current state of the
Apache REEF project and its place in the Hadoop ecosystem.

## Linux/OSS gain:

Originated at Microsoft Cloud and Information Services Lab and
recently Apache REEF (Sergiy Matusevych, Microsoft) - Recently graduated as a top level Apache project, Apache REEF is yet
another evidence of Microsoft commitment to support open
source. Battle tested in multiple already used in several large-scale production systems,
. Apache REEF fills an important niche in the Hadoop ecosystemstack, and
opens Hadoop to the community of C#/.NET developers. It allows developers to mix Java and .NET tasks in one distributed job, and, to our knowledge, is currently the only framework that provides .NET API to Apache YARN and Mesos resource managers.

## Speaker Bio:

Sergiy is a research engineer at Microsoft Cloud and Information
Services Lab, where he is building large scale distributed systems for
big data and machine learning. He is a committer to the Apache REEF
project. Prior to Microsoft, Sergiy worked as a data research engineer
at Yahoo! Research, and tried his hand in building machine learning
systems at several Silicon Valley startups. Sergiy is interested in
machine learning, data stream processing, and high performance
distributed systems. His previous speaking experience includes high
profile conferences like UAI, MLDS, and HadoopSummit conferences.