Lab 3: Chunked File Transfer

You will notice that this lab doesn’t have a Github repo associated with it… That’s because you will be checking the completed code into your Project 1 repo. Our goal for this lab is to build some of the DFS file transfer functionality.

You are responsible for building two components:

Let’s look at the functionality expected on both sides of this system.

Client

The client will take four command line arguments: a hostname, a port, a file, and a chunk size (in bytes). For example:

./client orion03 8888 ./suicide_squad.mkv 1000

This means that the client will read the file ‘suicide_squad.mkv’, split it into 1000-byte chunks, and then transfer the file to orion03:8888. Since most files won’t be evenly divisible by the chunk size, you will need to think about how to ensure the server side knows exactly how many bytes to expect.

Use protobufs to send file metadata. This will include the file name, number of chunks (and/or overall size), and a checksum of the original file so that the server can verify the transmission was a success. You can select any hash algorithm to generate your checksum.

In the end, the client will most likely:

  1. Create a protobuf message describing the incoming file
  2. Send the message to the server side so it can prepare to receive the file
  3. Read the file, splitting into chunks
  4. Transfer each chunk
  5. Wait for confirmation from the server that the file has been received

Server

The server’s role is opposite that of the client’s; it will listen on a port for incoming transfers, and when a client connects it will start receiving chunks and reassembling the file. Starting the server will require a port to listen on and a storage directory to place the reassembled files.

./server 8888 ./some/storage/dir

The server workflow looks like:

  1. Listen for client connections
  2. Read protobuf message with metadata
  3. Start reading file chunks and reassemble them into the original file
  4. Verify that the checksum of the reassembled file is correct

There are a variety of valid approaches for reassembling a file. Just remember that in the final version of your Project, the client will be doing the reassembling as multiple goroutines connect to StorageNodes and retrieve chunks.

Hints