Starter repository on GitHub: https://classroom.github.com/g/xszYnXHN

In this project, we will analyze a large dataset of user comments from popular news aggregation website Reddit. On Reddit, members submit content including news stories, articles, images, or videos and are also allowed to moderate the site via voting submissions up or down. The site is organized into a multitude of subreddits that specialize in particular types of content or discussion. For example, /r/politics covers US politics, and /r/technology focuses on tech news.

Similar to the content submissions, comments can also be voted up or down and several other metadata items are tracked. You’ll use these features to help you write Spark jobs that filter, aggregate, and glean insights from the dataset. You are encouraged to use Python (and Jupyter Notebooks) for this assignment, but you are given more leeway on using 3rd party libraries; for instance, since the dataset is in JSON format you may incorporate a JSON parser into your codebase.

Some students are familiar with Reddit while others may not be. As you explore the dataset, feel free to ask questions on Piazza or in your discussion groups. It’s also worth noting that since Reddit originated in the US, the comments and submissions will likely trend towards being US-centric. Additionally, certain demographics may be under- or over-represented by the dataset. You should keep factors such as these in mind as you perform your analysis.

Dataset Location

You can find the files in /bigdata/mmalensek/reddit/ on orion11 and orion12. This dataset was sourced from here.

Deliverables

You will submit two deliverables for Project 3:

A Jupyter notebook containing your code and answers to the questions below with your thoughts/opinions/analysis.
A project retrospective

As usual, some aspects of these questions are left up to your own interpretation. Occasionally there are no right/wrong answers, but you should be able to justify your approach.

Important: many of the questions are best answered with context. Think of it this way: if I ask you for a location, you may want to embed a map in your notebook or provide some statistics about it (population, nearby landmarks). Perhaps the question involves some obscure concept or subculture; in that case, a link to the appropriate Wikipedia article is useful. Combining different forms of media through data fusion can tell a compelling story (…just make sure the story isn’t misleading!). Failure to include necessary contextual information will result in deductions, regardless if you found the correct answer.

One final note: many of the questions below ask for you to find a specific user (or group of users). Be wary of bots or automated (non-human) accounts; perhaps the user that wrote the most comments in a particular time frame was just a bot that posts advertisements – in some cases, you will want to ignore these, so finding the top N users could be a better approach than finding the absolute top user.

Warm-up

[0.5 pt] How many unique subreddits are there?
[0.5 pt] Pick a subreddit. What user wrote the most comments in January of 2012? What was the user’s top three most-upvoted comments? Filter out bots or other types of automated posts.
[1 pt] Choose a day of significance to you (e.g., your birthday), and retrieve a 5% sample of the comments posted on this particular day across all 5 years of the dataset.
[1 pt] The number of comments posted per year will likely trend upward over time as more users join Reddit. However, the popularity of some subreddits may increase or decrease over time. Find An example of both.

Analysis

[1 pt] Screamers: It is well known that WRITING IN ALL CAPS ONLINE IS A SUBSTITUTE FOR SCREAMING… OR YELLING. *cough!*. (Or some might say it’s simply cruise control for cooooool). Write a job to find users that scream a lot, and provide a screamer score (a highly-technical metric that you will invent).
- For future reference (when we really want to get something off our chest), what are the top 5 subreddits for scream-y comments?
[3 pt] Readability: write a job that computes Gunning Fog Index and Flesch-Kincaid Readability (both reading ease and grade level) of user comments. Then:
- Choose a subreddit and plot the distribution of these scores.
- Choose two subreddits focused on similar topics but with different views, e.g., /r/apple and /r/android. Use these metrics to compare the populations from both.
[2 pt] Key Terms: calculate the TF-IDF for a given subreddit.
- Produce a Tag Cloud of the terms (note: this doesn’t have to be integrated into your code; simply including the image is enough).
[2 pt] Toxicity: using Sentiment Analysis, determine the top 5 positive subreddits and top 5 negative subreddits based on comment sentiment.
- Choose two subreddits focused on similar topics but with different views, e.g., /r/apple and /r/android. Compare the toxicity of the two.
[3 pt] Backstory: given a specific user, find out more about them: where they’re from, what things they like/dislike, and other data about their background (think of at least 2 more things to determine). Note that this should be somewhat automated; I should be able to give you a username and you’ll produce a backstory dataset for them (it may work better with some users than others). Provide a three sample user backstories in your report, written up as a narrative.
[4 pt] Now that you’ve found the answers to the questions above, design two of your own questions to answer. These should be sufficiently difficult, and you should be creative! You should start with a question, and then propose a predicted answer or hypothesis before writing a MapReduce job to answer it. If you come up with a particularly challenging question, it can count for two (ask first). Some ideas:
- Make predictions / classifications using ML models
- Visualization of related features. Your visualization should help tell a story.
- Clustering related users, comments, or subreddits
- Summary statistics: finding mins, maxes, standard deviations, or even correlations between variables to tell us something about a subreddit or multiple subreddits. For example, perhaps users that visit /r/technology also frequently visit /r/android.
- Friend graph: can you link together ‘related’ users based on some shared interest? Maybe several users visit the same collection of subreddits. The PageRank could come in handy here.

Final Analysis

You’ve had the opportunity to analyze two datasets thus far; now it’s time to analyze a dataset of your own. Find a dataset online and use Spark (or Hadoop) to analyze it. You should:

[0.5 pt] Describe the dataset
[0.5 pt] Outline the types of insights you hope to gain from it
[1 pt] Make hypotheses about what you might find
[4 pt] Design at least 3 “questions” (along the lines of those above) and answer them. Remember that presentation matters here. ML Models are a good choice for some of the datasets; you can describe what you’ll try to predict or classify and outline your experiences with various models.

Wrap-Up

[1 pt] Project retrospective

Hints and Tips

Some hints to remember while you’re analyzing the data:

Automated posts by bots, advertisers, etc. are not interesting. You should remove these unless they make sense to include in your analysis.
The dataset is not censored. We’re all adults here, but don’t put super offensive material in your writeup.
The dataset contains text-based identifiers that allow you to reconstruct threads/replies.
In most cases, you will want to remove “[deleted]” comments.
Comments may contain quotes of other users, which you may want to consider as separate from the comment itself. These are prefixed with > (like email replies)
In some cases, you may be able to get the correct answer without reading every line in the dataset; think about ways you can avoid reading data to speed up your computations.

Grading

Your grade is largely based on the quality of your report. I will deduct points if you violate any of the requirements listed in this document. I may also deduct points for poor design and/or formatting; please use good development practices, break your code into separate classes based on functionality, and include comments in your source where appropriate.

Changelog

11/18: Version 1.0 posted