Home

Welcome to Livy

Livy is a Service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or a RPC client library. Livy also simplifies the interaction between Spark from application servers, thus enabling the use of Spark for interactive web/mobile applications. Additional features include:

• Have long running Spark Contexts that can be used for multiple Spark jobs, by multiple clients
• Share cached RDDs or Dataframes across multiple jobs and clients
• Multiple Spark Contexts can be managed simultaneously, and the Spark Contexts run on the cluster (YARN/Mesos) instead of the Livy Server, for good fault tolerance and concurrency
• Jobs can be submitted as precompiled jars, snippets of code or via java/scala client API
• Ensure security via secure authenticated communication
• Apache-licensed, 100% open source.

More about Livy

This wiki is intended for developers interested in building Livy, learning about Livy's architecture and implementation or contributing - patches or otherwise - to the Livy project. For more information, check out the links in the sidebar, particularly Contributing to Livy and the developer mailing list.