CS686 Project 1: Distributed File System (v 1.4)

Due: October 18

In this project, you will build your own distributed file system (DFS) based on the technologies we’ve studied from Amazon, Google, and others. Your DFS will support multiple storage nodes responsible for managing data. Key features include:

• Parallel retrievals: large files will be split into multiple chunks. Client applications retrieve these chunks in parallel.
• Interoperability: the DFS will use Google Protocol Buffers to serialize messages. Do not use Java serialization. This allows other applications to easily implement your wire format.
• Fault tolerance: your system must be able to detect and withstand two concurrent storage node failures and continue operating normally. It will also be able to recover corrupted files.

Your implementation must be done in Java, and we will test it using the bass cluster here in the CS department. Communication between components must be implemented via sockets (not RMI or similar technologies) and you may not use any external libraries. The Java Development Kit has everything you need to complete this assignment.

Since this is a graduate-level class, you have leeway on how you design and implement your system. However, you should be able to explain your design decisions. Additionally, you must include the following components:

• Controller
• Storage Node
• Client

Version Control

To set up your submission repository on GitHub, visit: https://classroom.github.com/a/Yoknj2ce

In the spirit of versioning, I will update the version number at the top of this document every time a change is made and list any changes in the changelog below.

Controller

The Controller is responsible for managing resources in the system, somewhat like an HDFS NameNode. When a new storage node joins your DFS, the first thing it does is contact the Controller. The Controller manages a few data structures:

• A list of active nodes
• A list of files
• For each file, a list of its chunks and where they are located

When clients wish to store a new file, they will send a storage request to the controller, and it will reply with a list of destination storage nodes to send the chunks to. The Controller itself should never see any of the actual files, only their metadata.

The Controller is also responsible for detecting storage node failures and ensuring the system replication level is maintained. In your DFS, every chunk will be replicated twice for a total of 3 duplicate chunks. This means if a system goes down, you can re-route retrievals to a backup copy. You’ll also maintain the replication level by creating more copies in the event of a failure.

Storage Node

Storage nodes are responsible for storing and retrieving file chunks. When a chunk is stored, it will be checksummed so on-disk corruption can be detected.

Some messages that your storage node could accept (although you are certainly free to design your own):

• Store chunk [File name, Chunk Number, Chunk Data]
• Retrieve chunk [File name, Chunk Number]

One alternative is creating unique identifiers for each chunk, in which case you wouldn’t need the file name + chunk number combo. In that case, you’ll have to think about how you inform clients on how to reconstruct the file correctly.

Finally, the storage nodes will send a heartbeat to the controller periodically. The heartbeat includes chunk metadata to keep the Controller up to date, while also letting it know that the node is still alive. Heartbeats should be sent every 5 seconds and only include the latest changes at the node, not an entire list of its files. However, the Controller can also ask the storage nodes to send a complete file list (useful if the Controller failed and wants to rebuild its view of the system state). You can also include the amount of free space available at the node in your heartbeat messages so that the Controller has an idea of resource availability.

Client

The client’s main functions include:

• Breaking files into chunks, asking the Controller where to store them, and then sending them to the appropriate storage node(s). Note: Once the first chunk has been transferred to its destination storage node, that node will pass the chunk along in a pipeline fashion. The client should not send each chunk 3 times.
• Retrieving files in parallel. Each chunk in the file being retrieved will be requested and transferred on a separate thread. Once the chunks are retrieved, the file is reconstructed on the client machine.

The client will also be able to print out a list of files (retrieved from the Controller), and the total available disk space in the cluster (in GB).

Tips and Resources

• Log early, log often! You can use a logging framework or just simple println() calls, but you should print copious log messages. For example, if a StorageNode goes down, the Controller should probably print a message acknowledging so. This will help you debug your system and it also makes grading interviews go smoothly.
• Here’s an example of using Protocol Buffers for serialization and communication.
• Use the bass cluster (bass01 – bass24) to test your code in a distributed setting.
  • These nodes have the Protocol Buffers library installed as well as the protoc compiler.
  • To store your chunk data, use /home2/$(whoami), where $(whoami) is your user name.

Project Deliverables

This project will be worth 20% of your course grade (20 points). The deliverables include:

• [2 pts]: A brief design document (1 page). You may use UML diagrams, Vizio, OmniGraffle, or even pen and paper. This outlines:
  • Components of your DFS (this includes the components outlined above but might include other items that you think you’ll need)
  • Design decisions (how big the chunks should be, how you will place replicas, etc…)
  • Messages the components will use to communicate
• [6 pts]: Control node implementation:
  • [1] File list and chunk list
  • [1] Storage node list
  • [2] Detecting a node failure
  • [2] Replica maintenance
• [5 pts]: Storage node implementation:
  • [2] Storing chunks and checksums on the local disks
  • [2] Detecting (and recovering from) file corruption
  • [1] Heartbeat messages
• [5 pts]: Client implementation:
  • [2] Storing files (chunk creation, determining appropriate servers)
  • [2] Retrieving files in parallel
  • [1] Viewing the file list and available disk space
• [2 pts]: Project retrospective document

Note: your system must be able to support at least 10 active storage nodes. During grading, you will launch these components on the bass cluster.

Milestones

Here’s some milestones to guide your implementation:

• Week 1: design document complete, basic Protocol Buffer messaging implementation
• Week 2: client chunking functionality, controller data structures
• Week 3: storage node implementation, heartbeats, single-threaded client retrievals
• Week 4: failure detection and recovery, parallel retrievals
• Week 5: wrap-up, client file list/disk functions, retrospective

You are required to work alone on this project. However, you are certainly free to discuss the project with your peers. We will also conduct in-class lab sessions where you can:

• Get help, discuss issues, think about your design
• Demonstrate working functionality early to receive partial credit for completed milestones. If you didn’t understand a requirement or have an error in your logic, you will know early.

Grading

You will have a one-on-one interview and code review to grade your assignment. You will demonstrate the required functionality and walk through your design.

I will deduct points if you violate any of the requirements listed in this document — for example, using an unauthorized external library. I may also deduct points for poor design and/or formatting; please use good development practices, break your code into separate classes based on functionality, and include comments in your source where appropriate.

Changelog

• 10/17: Added link to project retrospective document.
• 10/4: Added user port assignment list.
• 9/7: Added a small hint about storing files on the bass cluster.
• 9/6: Added another bullet to the design document grade breakdown and added the due date: October 6.
• 9/6: Added note about pipelining storage requests; client should only send each chunk once.
• 8/31: Added tips and resources section
• 8/30: Version 1.0 posted