CS1951a: Data Science

Overview, Vision, Goals, and Hypotheses

Spotify has millions of playlists, constructed by algorithms, teams of experts, and users just like you and I. Have you ever wondered how Spotify generates their playlists? Have you ever wished there was a magical button you could press to do all the work of making your own playlists?

In this data science project, completed for CS1951A Data Science at Brown University, four students worked together on bringing this dream to life. At the start of the Spring 2019 semester, we stumbled upon an online data set of 4000 Spotify user-generated playlists, and set the vision of being able to build our own playlist recommendation program. Initially, we pursued this idea and attempted to implement the idea of using sentiment analysis to group songs according to the titles of the playlists they belonged to. However, along this road, we encountered a dead end, and felt completely lost.

Luckily, we were able to pivot directions. We found that through the Spotify API, we were able to access a set of thirteen audio features for every song on Spotify. With this newfound database of knowledge and the excitement of new possibilities, we launched two different efforts to realize our goal.

Firstly, we wrote a data collection program to obtain all the audio feature data for the songs in our set of 4000 user-generated playlists, generating a database (.db) of each song with information about it’s features. We made this design decision after iterating through other data structures (2D arrays, JSON), and found that it would allow for the most efficient manipulation through using sqlite3 and SQL. Choosing four audio features, tempo, danceability, energy, and valence, and essentially having a set of data points with four dimensions, we were able to run K-Means clustering to group the songs into 7 clusters. This effectively divided the songs into 7 unlabelled “topics”. However, we wanted to take it a step further. In order to investigate which of the audio features were the features best representative of the variation in the data (essentially, best measures of similarity), we chose to run K-Means instead on every combination of pairs of audio features (e.g. Tempo + Danceability, Energy + Valence, etc). We created clustering scatter plots to view the resulting clusters, and created a bar plot showing the clustering errors from each combination. We looked for the lowest clustering error, which would suggest to us that that combination of features was best representative of the data.

Secondly, we decided to use the skills we gained from our Deep Learning Homework 8 to implement a version of the word2vec algorithm that was used. Looking back at the 4000 user-generated playlists, we treated each playlist like a “sentence”, and each song in the playlist like a “word”. Implementing song vectorization, we essentially were measuring how often songs appear in the same playlist together. From creating the song vectorizations of how often songs appear in the same context together, we were then able to distances between song vectors using the cosine similarity metric. This allowed us to then run K-Means clustering on our set of song vectors, to again group the songs into 6 clusters of general “topics”, but then also increase the amount of clusters to 1,500, in order to generate much smaller playlists of 10-20 songs.

Ultimately, our hypotheses/goals were to:

Determine which audio features were most important in defining songs. We estimated, from our personal experience and interaction with the music we listen to, that the two most important audio features that define songs are tempo and valence.
Be able to generate our own algorithmic playlists usings two means:
1. K-Means clustering on audio features
2. Song vectorization and K-means clustering on song vectors

We hypothesized that the playlists generated from the song vectorization of

previous user-generated playlists would create better, more fitting groupings of songs,

than clustering by audio features.

Our Data: Spotify user-generated playlists & Spotify API

For our project, we used two bodies of data. Firstly, we used a dataset of 4000 user-generated Spotify playlists which consisted of a JSON object that contains a dictionary of playlists. Secondly, we used the Spotify API to collect a set of audio features for every song in those 4000 playlists.

We initially thought that 4000 playlists might be a little too small for our purposes, but we discovered that these playlists make up 75,833 unique songs. This was, if anything, more than enough for both our ML models (song vectorization, and clustering by audio features). We actually were not able to, and eventually chose not to, run the models on the entire song corpus, and instead truncate our data set, for two reasons:

When we tried to perform song vectorization on the entire dataset, all of our computers ran out of RAM memory when constructing the lookup table. Specifically, the method that constructs the lookup table in utils_playlists.py would construct the table but not be able to return it because it was a huge NumPy array. We were going to change our program up so that the construction of the lookup_table is not done in a function, but we chose not to because of the second reason below.
We realized that the frequency by which each of the unique songs appears within playlists decreases dramatically when we increase the sample size that we truncate our dataset by. We truncate the dataset by choosing the top most popular N songs where popularity is determined by how frequently do these songs appear in unique playlists of the dataset. We found that for n = 5000, the least popular songs appear with frequency 9; for n = 15,000, the least popular songs appear with frequency 3; and for n = 20,000, the least popular songs appear with frequency 2. We then faced the design decision of either:
- Choosing a larger N, which would have produced larger but less dense (compact) K-Means clusters since we would have more songs but they will be less similar (because they appear less frequently). This covers a larger portion of our data at the expense of playlist accuracy.
- Choosing a lower N, which would have produced smaller but more dense (compact) clusters since we would have less songs but they will be more similar (because they appear more frequently). This sacrifices a big portion of the data but give us much more accurate playlists.

Eventually, we decided to go with the second option so we can have better visualizations and more accurate playlists. We talked to Elie and she told us that this design choice is very common among data scientists and that it is often better to sample the head of the distribution because we have more confidence in it.

We were able to find exactly what we wanted in the dataset. We only needed playlist titles, track titles, and track URIs from the main dataset. The URIs were later used to make API calls to the Spotify server to retrieve a predefined set of audio features for each of the songs in our dataset.

Collecting the data was by far the most time-consuming part of our project. We wrote a script (data_collector.py) to collect the data that we needed and to dump it in a .db file and a .txt file. The data collection process was as follows (code referred to here is from data_collector.py):

We loaded the JSON file file into a python list and initialized a Spotipy object which we used for API calls later. This was done in the “__init__” method.
We created a main data corpus that was later used to construct our two databases used for clustering and vectorization. This main corpus was a table in which each row represented a playlist and each column represented one of the 11 features of that playlist. Most notably, the 1st column was the playlist name and the 8th column was a list of the tracks that comprise said playlist. The 8th column was a special one in that it was its own list of track features. That is, the tracks feature of each playlist was another list in which each row represented a track in that playlist and each column represented one of the 8 features of a track. This was done in the “construct_playlist_to_tracks” method.
We constructed a mapping from playlist titles to track titles within that playlist. This was a list of lists in which the i-th inner list was representative of the i-th playlist, and the j-th element within that list was representative of the j-th track within that playlist. This was done in the “construct_playlist_to_tracks” method, and its output mapping was dumped to a .txt file using the “pickle” library.
We constructed a mapping from track titles to values of its audio features (tempo – valence – danceability – energy). This was a dictionary in which keys were unique track titles and values were lists of these tracks’ audio features. This part of the data collection took a large portion of our time because it needed to make 75,833 API calls to the Spotify server which was way (way) beyond the imposed limit. We tried to batch API calls by 50 track URIs at a time (which is the server’s limit on URIs to be considered as one API call), but we later noticed that there were discrepancies resulting from the server returning audio features in an order different to how we sent the track URIs. As a result, we decided to make API calls one-at-a-time, and then store that result in a database so that we do not have to make the calls again. This took quite some time but we were eventually able to collect all of the needed data and store it in a database. We realized at a later point in the project that we only saved the track titles but not the artist names in our database, but we did not want to go through the API calls problem once again. To fix this, we wrote a script that constructs another database binding track titles to artist names. We then used sqlite3 to query both databases and join tables on track titles in order to obtain our desired table in which we also have artist names. We then used a vocabulary containing the titles of the most popular 5,000 track to truncate the database to include only the most frequent tracks in our dataset. This was done mostly in the “construct_song_to_features” and “save_data” methods, but involved some extra local scripts and data manipulation using sqlite3 and SQL command.

The end results of our data collections were the following two files:

tracks2features_full.csv: contains a table that has the track title, artist name, valence, tempo, danceability, energy, and speechiness of the most popular 5,000 songs in our dataset.
playlist2tracks_full.txt: contains a pickle dump which, when loaded, produced a list of lists in which the i-th inner list is the i-th playlist in our dataset, and the j-th element within that inner list is the j-th (track title, artist name) tuple within that playlist. This contains all of the playlists inner dataset and is later truncated in sparse_playlists.py

Notes:

We chose the set (valence, tempo, danceability, energy, and speechiness) as our audio features by talking to some of our musician friends in the music department. They suggested that this set is the most representative of a song out of the 13 pre-defined audio features in the Spotify API.
This is the main link from which we got our main dataset. Originally it was sliced into 4 slices of 1,000 playlists each, but we merged it into one file containing all 4,000 playlists. We will not be handing-in the merged file since it would be too big to hand-in, but it will be uploaded onto our Google Drive folder. The link from which were obtained our data slices is: https://github.com/vaslnk/Spotify-Song-Recommendation-ML/tree/master/data?fbclid=IwAR30mCJqGeehmMyhwkK26WvK5grtOB_ciP-Rjlpwa-2avySKdSNtu4c1-qs

FUN FACT: the dot at the bottom left corner is the song “Super Mario Bros – Original” with track URI “spotify:track:4DG4um6R0wx4WrqhhACc5g”. We discovered, through our API calls, that this song has 0 valence, 0 danceability, 0 tempo, and 0.331 energy (you can check that using the track URI at: https://developer.spotify.com/console/get-audio-features-track/?id=spotify%3Atrack%3A4DG4um6R0wx4WrqhhACc5g). We decided to use this point anyway since it has an energy value, and we were interested in how it could affect our clustering models. We found that it did not drastically make a difference to include it.

Method 1: Song Vectorization and K-Means Clustering

Our goal was to analyse song similarity based on how often songs appeared together in the same playlist. Using the skills we learnt in the DL assignment, we treated each song as a “word” and each playlist as a “sentence” in order to implement a similar sparse_vecs structure. Looking into our dataset, since many songs only appeared in a playlist once, we inferred that running K-Means clustering on our entire dataset would result in many outlier points of rare songs that are far away from the main clusters of frequent songs, while the playlists we generated wouldn’t be as accurate based on user-sentiment. Because of this, along with consultation from Ellie, we decided to only analyse the 5000 most popular songs, where each song appeared at least in 9 playlists, in order to optimize the tradeoff between having a large enough subset to have reliable results and generating accurate clusters and playlists.

To model these 5000 songs, we created a 2D array size 5000 x 5000, with each row index being a song id, which maps to the song title in our vocab dictionary, and the value being a vector of length 5000 counting the number of times each song appears with the current indexed song in the same playlist, which we calculate in linear time by looping through each song in each playlist once. After constructing this lookup table, we calculate the cosine similarity between each song pair, storing the result in a 2D array of 5000 x 5000 where each row is now a vector showing the similarity between each song and the indexed song. Now that we have 5000 known “features” describing each song, we used TSNE to dimensionally reduce the 5000 features into 2, in order to easily cluster and visualize our results while maintaining the most important variance in our data-points. To determine the optimum number of clusters for K-Means, we used the following elbow-point graph:

As shown from above, having n=6 clusters seemed to be the most optimum point since it was the last inflexion point where the subsequent decrease in gradient was the smallest, so we determined this was our “elbow” point. Using K Means with 6 clusters, we had the following visualization:

Most of our data seems to cluster very well, except for a few outliers in our red and green clusters, although it was difficult to label each cluster since each cluster had around 1000 songs spanning across many genres and time periods. However, when running K-Means on the entire dataset with 500 clusters, we were able to generate playlists with an average of 10 songs each, which we present in our conclusion with more analytical detail. We also tried generating playlists for all the songs in our dataset (75,833 unique), but was unable to do so due to limits in RAM.

Method 2: Audio Feature Clustering

We also wanted to use machine learning techniques to analyze song similarity using the audio features provided by the Spotify API. These audio features all have numerical values, making them a good fit for K-means. Spotify provides 13 audio features in total, but after interviewing and speaking to friends in the music department, we settled on five initial audio features to cluster on: valence, tempo, danceability, energy, and speechiness. However, after conducting the two approaches below, we realized that speechiness was not an optimal determinant of similarity. The clustering graphs with speechiness included were sparse and not tightly grouped at all, and clustering error was strangely low. We took a look through the dataset, and after all of this evidence, we concluded that speechiness wasn’t a good measure of variation in the data because most songs had speechiness values that leaned heavily towards zero. We thus narrowed down the total number of audio features to just four: valence, tempo, danceability, and energy.

K-Means on all four audio features

Our first approach involving audio feature clustering was to cluster with all four features, which is done in final_kmeans.py. To maintain consistency with the the song vectorization method, we also used the top 5,000 songs as our data points. Before proceeding, we normalized the audio features, as some of the audio features had different scales from one another. Our method for doing this was to find the maximum value for each audio feature, and then to divide the value for each data point by that maximum value. This resulted in a scale of zero to one for each audio feature, allowing us to maintain even more consistency. We produced a numpy array containing the all of normalized feature values for each data point, and we fed those into sci-kit learn to the cluster IDs for each data point. We concatenated those cluster IDs to the initial numpy array so that each data point now had its associated cluster ID as well, making it ready for visualization. However, because there are four features and therefore, four axis dimensions, our clusters were impossible to visualize in their original form. To remedy this, we used the hypertools library to perform PCA dimensionality reduction to three dimensions before plotting the clusters.

Clustering based on all of four our features created clean clusters, as seen in the visualization above. The above visualization has seven clusters, because after looking at the elbow point plot, seven was where the last inflection point was, signifying the last major change in rate of error.

Because of the dimensionality reduction and because of the unsupervised nature of K-means, it is unclear what the axes represent and what specific “topics” that the clusters themselves actually represent. However, the fact that the clusters are so clean is a positive sign, and the main takeaway from this approach is that these four audio features can effectively predict similarity for songs together.

Audio feature pairs

Our second, and more important, approach involving audio feature clustering was to cluster with different combinations of the two of the four features. We do this in final_audio_pairs.py. There were six such combinations in total. As with the “all four features” analysis, we normalized the audio feature values. We produced six numpy arrays containing the different combinations of features for each data point, and we fed those into sci-kit learn to generate the cluster centers and the cluster IDs for each data point. We then used matplotlib to produce visualizations of the clusters with each combination of audio features, where each axis represents a measure of the audio feature.

We also decided to create a visualization for the errors of the six different audio feature pairs to better determine which pair produced the lowest error with audio_errors.py. Similar to above, we produced six numpy arrays containing the different combinations of features and fed those into sci-kit learn’s fit_predict. We then retrieved scores for those combinations and used those to calculate the error associated with each audio feature pair. Finally, we used the bar chart in the matplotlib library to visualize the errors.

As seen above, the combination of tempo and danceability produced the lowest error, suggesting that those two features were the best features for determining the variation between songs. This contradicted our initial hypothesis that valence and tempo were the best predictors of similarity.

Generating Playlists

To test our second hypothesis of whether song vectorization or audio featuring clustering was better at predicting song similarity, we also generated playlists with audio feature clustering in audio_playlist_generation.py. To maintain consistency, we also clustered into 500 clusters over the top 5,000 songs, as with song vectorization.

Conclusions

We’ve come a long way since the beginning of our project, and we are proud to present the results that we’ve been able to obtain.

Firstly, we determined through K-Means clustering on various audio features of songs, that through looking at the clustering errors, the pair of audio features that results in the lowest error is Tempo & Danceability. This breaks our original hypothesis, but also follows along intuition. However, it is important to note that there are limitations to this conclusion. In particular, all the songs that were analyzed were a part of the 4000 user-generated playlists, which is only a very small sample of the many millions of playlists on Spotify. Furthermore, this set of songs from these 4000 playlists may or may not be representative of the general population of Spotify users, and even more so, the general population of Spotify users may not represent the musical tastes of all human music-enjoyers. Thus, it is important to keep in mind the source of our data in interpreting this outcome. This applies also to our secondary results.

Secondly, through the method of K-Means on audio features and K-Means on song vectorizations, we were able to use two separate methods of algorithmically generating playlists. We hypothesized that the playlists generated by the song vectorization method would group songs that were more “similar”, fitting our human intuition. In order to visualize the many playlists that were created (which can also be accessed through our submission), we created album covers.

Here are some album covers generated from playlists using the song vectorization method:

Because these playlists are unnamed, as they were algorithmically generated, we labelled them ourselves. Above, we named the playlist on the left “pop”, and the playlist on the right “rap”. Below, we named the playlist on the left “throwbacks”, and the playlist on the right “christmas”.

Here we can clearly see that a lot of these playlists work very very well, which was a very exciting result for us. All these groupings make a lot of sense, since playlists in this method were generated depending on how often songs showed up in the same user-generated playlists together. Of course songs from the early 2000s showed up more often than not in the same playlists together, likely titled “throwback” for college aged students like us.

Alternatively, here are some playlists generated by the clustering using audio features:

In these examples of playlists generated from audio feature clustering, it is unclear whether or not there is a clear correlation between the songs. There are some surprising groupings, such as on the bottom right, which groups together country music, disney pop songs, Christian worship songs, and alternative metal. Unfortunately, this was a pattern for most of the playlists generated by the audio features grouping – songs often times did not fit into the same “vibe” or genre, creating groupings of songs that were simply clashing. Perhaps, however, many of these songs that are grouped using this method actually share similar quantitative features such as the tempo of the song, or the mood (valence) of the song, but regardless, songs are defined by much more than a set of quantifying numbers.

All in all, we were very excited to be able to come so far in our project, especially since we hit what felt like such a major roadblock around midway through the project. We enjoyed the process, developed and strengthened many skills that we had learned in class, and gained a greater appreciation for the complexity of how Spotify generates their countless playlists and is able to constantly suggest new, fitting music for its users. We were also very proud to be able to create our project poster, and are excited to continue using our skills in data collection, machine learning, and data analysis in the future. Special thanks to Anna Nakai, our team TA mentor, and Professor Ellie Pavlick, who always pulled through and advised us in the thickest of moments, when our backs were against the wall. Thank you for taking the time to read through our blog, and we wish you the best!

Cheers,

Jason Chan, Nazem Aldroubi, Martin Chu, Kevin Xiang

CS1951a: Data Science

Final Blog Post

Big Boi Blog Post #1