Synthetic File System

The student team is to develop a new file system that provides a unified view ona pre-configured set of existing file systems. The new virtual file system is to:
●Be implemented as a Linux FUSE (filesystem in user space) module

●Be POSIX compliant; initially, only read-only functions are needed

●Integrate NTFS, ext4, and S3 object stores

●Be able to filter which files are shown using meta-data

●Be configured from a configuration file during startup time





Test cases are:

●Browsing using an existing Linux file system browser (e.g. Nautilus)

●Using cp to copy from the new virtual file system



amos-ss2021-synthetic-file-system Bug · Requested Features

Table of Contents

1. About The Project
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. License
6. Contact

About The Project

This project implements a so called FUSE, a filesystem in userspace, in order to give a unified filesystem view on data in a Metadatahub database.
The FUSE also allows for filtering the files it will show according to their metadata

Product Vision

The rapidly growing mass of data requires further refinements and new technologies in order to find the right data in this deluge of information. The creation and management of metadata is decisive for representing the content of stored objects and files. This metadata is held in a database for instant retrieval. Lists can be constructed from these databases to find files and objects in general, but they do not yet afford access to the real data. For this reason, a synthetic file system is considerably useful. A synthetic file system enables to access data, chosen by queries in the database of metadata. Retrieval of project-related data is done through a distributed database and a virtual file system that permits a single namespace for all relevant data designated by their metadata.

Project Mission

Due to the current Corona pandemic, as much data as possible is to be analyzed and evaluated with the help of artificial intelligence (AI). This requires a central intelligence to collect and interpret all accessible data distributed over several facilities. The key issue is that the data is organized and saved in different systems according to different storage types, structures, formats and criteria. The task, or rather mission is now to first make the data obtainable or readable via a uniform mechanism within the project time-frame. This would have the benefit of not having to adapt each application to the different memory types. The synthetic file system is a so-called adapter for each type of memory, so that a unified namespace can be formed from it.

---

Getting Started

Prerequisites

• git
• docker: follow instructions for your platform (Only needed if you want to try the fuse in the docker)
• docker-compose (Only needed if you want to try the fuse in the docker)
• Metadatahub

Installation

1. Clone the amos-ss2021-synthetic-file-system

   git clone https://github.com/amosproj/amos-ss2021-synthetic-file-system.git

2. If you want to try the FUSE in a docker environment, build it with

   cd amos-ss2021-synthetic-file-system
   docker-compose up --build 

   For subsequent uses you can just run

   docker-compose up

   Then you can connect to the docker from a different shell via

   docker exec -it synthetic-file-system tmux

   where tmux can also be replaced by zsh or bash, depending on your preferences.

3. This fileystem will be mounted under ~/fuse_mount. Create this folder for later use:

   mkdir ~/fuse_mount

---

Usage

Basics

1. SFS pulls its information from a running Metadatahub service. For more information or documentation refer to the vendor.

2. To run the SFS without docker, simply run

    python3 -m src mount_point

   where mount_point is an empty folder you specify by yourself.

3. Warning: SFS with docker is bugged in the current release version and will be resolved soon

4. Attention when start the FUSE using docker: Since the FUSE blocks the current terminal, a new terminal in the docker has to be opened. For this you can just open a new terminal on the host and connect it again to the docker via docker exec -it synthetic-file-system tmux , or use tmux in the docker to open a new terminal (ctrl+b -> ctrl+%). For more information please refer to the tmux documentation.

5. Traverse the virtual filesystem via a terminal, or via any file browser like nautilus. For docker: The docker container is configured to support X-forwarding, so any UI program opened on the docker will be forwarded to the host. So to traverse the filesystem using a file browser under the docker, just run

    nautilus

   from a terminal that is connected to the docker.

Configuration

SFS enables the filtering of files via their metadata. For this, a config file config/config.graphql is used. When starting SFS, all the filters will be read from this file and applied. More information regarding graphl syntax can be found here.

NOTE
The way the dynamic configuration works, it is needed, that the config file is written to and closed properly, just for example pressing ctrl+s won't always work. So when VIM for example, close the file with :wq or :x.

---

Roadmap

See the open issues for a list of proposed features (and known issues).

License

Distributed under the MIT License. See LICENSE for more information.

Contact

AMOS PROJECT - [email protected]

Industry Partner - GRAU Data

Project Link: https://github.com/amosproj/amos-ss2021-synthetic-file-system