Distributed File system

This project is distributed file system based on FUSE. Currently the cluster contains only two servers, one primary and one secondary, but it could be extended to more.

How to run the program

Before compiling the project, make sure the environment is set-up.

sudo apt update 
sudo apt dist-upgrade 
sudo apt install build-essential pkg-config libfuse-dev rpcbind

This repo has provided Makefile, but several places needs to be updated.

make header compiles and updates the RPC header and stab files.

make all compiles client-side and server-side executable file tweakfs and fuse_rpc_server respectively.

make create generates the rootdir and mountdir automatically, which is the pre-requisite for the program.

make debug needs a little update: the last two arguments are IP addresses for primary and secondary server.

make unmount unmount the mountdir off rootdir.

make clean cleans all generated object files and rootdir, mountdir.

Additionally, on server-side, I did a hard code of secondary server IP address, connect_serverwhich needs to be corrected.

System configuration file /etc/hosts should also appended the target server IP. Example:

127.0.0.1       localhost
127.0.1.1       ubuntu

# For RPC.
67.159.88.71 [email protected]
10.148.54.200 [email protected]

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Implementation details

1. On client side, every file system operation will be transmitted to server side via RPC. In this project, I select Sun RPC. Client side always tries to connect primary server first, then secondary server if fails.
2. .x file is the communication protocal for RPC. Naming pattern:

• client-to-server message structs are named as function_arg, server-to-client message structs are named as function_ret.
• Every server-to-client response includes a data member ret, it returns 0 if RPC succeeds, otherwise it'd be assigned -errno.
• For r/w requests, a len member is included, which indicates the actual bytes completes.
• Every *_arg has a data member called ip, which is used to differentiate seperate path for distinct clients. To create path at server side, it's represented as a unsigned integer.

3. Server handles every RPC requests from client side. For primary server, several updating operation needs to be completed on secondary server for replication and availability, say, write, mkdir, rmdir, etc. It first transmit these requests to secondary server, and handle the request whatever the second one succeeds or fails. Other read-only requests will only be handled on primary server.
4. Primary server goes through the "handle flow" of opendir - readdir - releasedir, open - read - release, so it could use the fd acquired from previous operations, while the secondary server have to rely on full path.
5. All operations are executed in the order of:

• transmit request to primary server, if possible
• check rwx validility, if needed
• lock the file or directory, if needed
• execute the operation
• unlock the locked resource
• set return value, and send back response back to client via RPC

6. Server stores all files under the root directory /DFS, and create it at the very beginning it doesn't exist. All files are considered visible by all user mounted, but other operations, like updating, removing, reading needs further authentication.
7. About access authentication:

• file open is checked by read and write bit
• directory listing is checked by read bit, open is checked by execute bit
• file creation, rename and deletion need the rwx bits of parent directory, related operations include operation: unlink, mkdir, rmdir, mknod, rename, symlink, link
• chown needs root, chmod needs owner, utime needs owner

8. Locking mechanism:

• add shared lock for file read and directory read
• add exclusive lock for file write
• add exclusive lock for directory mutation, eg: rename, unlink, etc

9. About server recovery, we have provided a python script which does cold restoration. Eg, you could simply sudo python rsync.py [email protected]:/9962309 /

Special handling

1. chown needs root identity, users calling it will get a "permission deny" error message.
2. /DFS is a special root directory in Ddistributed File System. Anyone can create file or directory inside of it, but only owner of the file can delete or rename.

Known bugs

1. We have handled the cases where primary server or secondary one cannot be reached. But if the client goes down, fd won't be released, thus system resource is leaked.
2. symlink and link operations are not yet thread-safe.
3. server failover test doesn't pass; currently it can only handle the case when server program is killed, and machine is still alive; it doesn't go well when secondary server is rebooted.

Reference

RPC:

http://tharikasblogs.blogspot.com/p/how-to-write-simple-rpc-programme.html

https://docs.oracle.com/cd/E19683-01/816-1435/6m7rrfn9k/index.html