Lithops is a Python multi-cloud serverless computing framework. It allows to run unmodified local python code at massive scale in the main serverless computing platforms.
Lithops delivers the user’s code into the cloud without requiring knowledge of how it is deployed and run. Moreover, its multicloud-agnostic architecture ensures portability across cloud providers, overcoming vendor lock-in.
Lithops provides great value for data-intensive applications like Big Data analytics and embarrassingly parallel jobs.
It is specially suited for highly-parallel programs with little or no need for communication between processes.
Examples of applications that run with Lithops include Monte Carlo simulations, deep learning and machine learning processes, metabolomics computations, and geospatial analytics, to name a few.
Lithops facilitates consuming data from object storage (like AWS S3, GCP Storage or IBM Cloud Object Storage) by providing automatic partitioning and data discovery for common data formats like CSV.
Lithops abstracts away the underlying cloud-specific APIs for accessing storage and provides an intuitive and easy to use interface to process high volumes of data.
Lithops is available for Python 3.6 and up. Install it using pip:
pip install -U lithops
You're ready to execute a simple example!
from lithops import FunctionExecutor
def hello(name):
return 'Hello {}!'.format(name)
with FunctionExecutor() as fexec:
fut = fexec.call_async(hello, 'World')
print(fut.result())Lithops provides an extensible backend architecture that is designed to work with different compute and storage services available on Cloud providers and on-premise backends.
In this sense, you can code your application in Python and run it unmodified wherever your data is located at: IBM Cloud, AWS, Azure, Google Cloud and Alibaba Aliyun...
- Simplify the developer experience with OpenShift for Big Data processing by using Lithops framework
- Speed-up your Python applications using Lithops and Serverless Cloud resources
- Serverless Without Constraints
- Lithops, a Multi-cloud Serverless Programming Framework
- CNCF Webinar - Toward Hybrid Cloud Serverless Transparency with Lithops Framework
- Using Serverless to Run Your Python Code on 1000 Cores by Changing Two Lines of Code
- Decoding dark molecular matter in spatial metabolomics with IBM Cloud Functions
- Your easy move to serverless computing and radically simplified data processing Strata Data Conference, NY 2019
- Speed up data pre-processing with Lithops in deep learning
- Predicting the future with Monte Carlo simulations over IBM Cloud Functions
- Process large data sets at massive scale with Lithops over IBM Cloud Functions
- Industrial project in Technion on Lithops
- Outsourcing Data Processing Jobs with Lithops - IEEE Transactions on Cloud Computing 2022
- Towards Multicloud Access Transparency in Serverless Computing - IEEE Software 2021
- Primula: a Practical Shuffle/Sort Operator for Serverless Computing - ACM/IFIP International Middleware Conference 2020. See Primula presentation here
- Bringing scaling transparency to Proteomics applications with serverless computing - 6th International Workshop on Serverless Computing (WoSC6) 2020. See Workshop presentation here
- Serverless data analytics in the IBM Cloud - ACM/IFIP International Middleware Conference 2018
.. toctree:: :hidden: self
.. toctree:: :hidden: :maxdepth: 0 :caption: Overview of Lithops source/design.rst source/comparing_lithops.rst source/supported_clouds.rst source/execution_modes.rst source/cli.rst
.. toctree:: :hidden: :maxdepth: 0 :caption: Install and Configure Lithops source/install_lithops.rst source/configuration.rst source/compute_backends.rst source/storage_backends.rst
.. toctree:: :hidden: :maxdepth: 0 :caption: Lithops Compute API source/api_futures.rst source/functions.md source/notebooks/function_chaining.ipynb source/api_stats.rst
.. toctree:: :hidden: :maxdepth: 0 :caption: Lithops Storage API source/api_storage.md
.. toctree:: :hidden: :maxdepth: 0 :caption: Data Processing source/data_processing.md source/data_partitioning.md
.. toctree:: :hidden: :maxdepth: 0 :caption: Integrations source/api_multiprocessing.rst source/api_storage_os.md source/sklearn_joblib.rst source/airflow.rst
.. toctree:: :hidden: :maxdepth: 0 :caption: Advanced Features source/worker_granularity.rst source/monitoring.rst Custom Runtime <https://github.com/lithops-cloud/lithops/tree/master/runtime> source/customized_runtime.rst
.. toctree:: :hidden: :maxdepth: 0 :caption: Experimental Features source/metrics.rst source/dso.rst
.. toctree:: :hidden: :maxdepth: 0 :caption: Developer Guide Applications <https://github.com/lithops-cloud/applications> source/contributing.rst source/testing.rst Changelog <https://github.com/lithops-cloud/lithops/blob/master/CHANGELOG.md>