Motivation
I am a big fan of Google Cloud Platform, especially I love its data warehouse implementation called BigQuery. In summer of 2016 Github and Google made the open-source data available for everyone in BigQuery, here are the mind boggling numbers:

This 3TB+ dataset comprises the largest released source of GitHub activity to date. It contains a full snapshot of the content of more than 2.8 million open source GitHub repositories including more than 145 million unique commits, over 2 billion different file paths, and the contents of the latest revision for 163 million files, all of which are searchable with regular expressions.

I have since then never tired of exploring these data, revealing interesting patterns or extreme samples and publishing articles about my findings.
In December of 2016 Google team has woken up my “researcher within” again — they’ve added Stack Overflow’s history of questions and answers to the collection of public datasets on BigQuery. In practice that means that the most popular programming chat in the world is now can be analyzed with the power of Google Cloud Platform, for example one can run the sentiment analysis on the Stack Overflow data and find out that Python developers post the lowest percent of negative comments overall! What excites me the most though is the ability to join the Stack Overflow data with other publicly available data sets. For example one can try to find out whether the weather can affect the probability of a Stack Overflow question to be answered by using the data from NOAA dataset(I am actually going to conduct this research soon).
In the introduction to the Stack Overflow data availability Felipe Hoffa provided the sample of joining Github and Stack Overflow data to find out which are the most referenced Stack Overflow questions in the GitHub code — specifically, Javascript. It gripped my attention because I noticed a couple of limitations:
* The query searches only for stackoverflow.com/questions/([0–9]+)/ pattern in the source code. However, there are alternative forms of referencing questions : it could a short form stackoverflow.com/q/([0–9]+)/ and it could be the direct reference to one of the answers, like stackoverflow.com/answers/([0–9]+)/
* The query deals only with JavaScript sources, but there are plenty of other programming languages.

So, I set out to build the catalog of the stack overflow questions referenced in the GitHub sources for popular programming languages.

Getting the data
Step 1 Finding lines of code in Github Sources that have references to StackOverflow questions or answers. contents_top_repos_top_langs table that keeps contents for the top languages from the top repos was kindly provided by Felipe Hoffa

The result has been saved in the new table called so_ref_top_repos_top_langs which contains the rows like

Step 2 Join the result with the StackOverflow data. The query should handle both of the questions id’s and answers id’s extracted from the source code

The result contains the rows the look like

There were roughly 31K records like this one, the next question was on how to visualize them.

Building the web-site
First of all I moved the resulting data to the SQLite database by creating separate table for each programming language. Then I built the [web-site](http://sociting.biz) that allows to navigate through the data by switching between the languages and jump to the Github source code to check how the information from the questions/answers was applied in the specific scenarios. I also caught this opportunity to play with ASP.NET Core and implement the web-site on my Macbook Pro, without Windows being involved. The resulting application uses the cross platform ASP MVC Web API on the back end and react+redux on the front end. The source code is fully available in the Github repo.

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising &sponsorship opportunities.

To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!

One More Analysis of GitHub and StackOverflow Data with Google BigQuery was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

If a prehistoric developer had been frozen in 2010 and then thawed out in 2016, he would discover the whole new world. Imagine that he met a modern react.js ninja who tried to explain the way we are using to implement login forms in 2016.

React components and containers. We now wrap login forms in the React Components in order to integrate Login function into React apps. The sample below was copied from React Redux Universal Starter Kit project.

And it’s not enough! Some guy convinced us to keep apart presentational and container components, so we bring this separation even for such a simple function as Login. It’s funny that there is no single recipe but a wide range of solutions for doing that, here are few examples:

• Using redux-form which is the decorator that wraps a form in a Higher Order Component (whatever it means), like it’s done in the full code for the Login Form shown above.
• Introducing a container for Login form, like the one in relax project:

• Adding the “handler” of login form, then wrapping it with the help of react-redux, just like it’s done in React Redux Universal (isomorphic) bundle project:

I guess that at this point the facial expression of the prehistoric developer would be close to what is shown in the Toy Story picture, but there are more things to explain.

Single Page Apps and Reactive State. We don’t let browser send the login form by using its native capabilities. Not anymore. We are obsessed with the idea of hosting the whole application on a single page and simultaneously pretending that it contains multiple pages by having total control over browser’s history which requires manual submitting of a login form’s data. Therefore we intercept the submit event of a form and pass the entered username and password further to the login request, but in order to obtain these values we don’t refer to the corresponding DOM elements(as we would normally do with jQuery). No sir. It is indeed possible, but the React documentation calls it “escape hatch” and suggest not to overuse it. Instead we used so-called “controlled text inputs” that on every keystroke raise the change event whose handler updates the state object which is used to re-paint the input box. The snippet of code below copied from gelatinous-toboggan project:

It seems like we overwrite the native browser’s implementation of the input element‘s behavior time and again(One more example). Though some delinquents bypass this practice:

Prehistoric developer: do you have the time machine in 2016 so that I can escape from this madness?

Don’t get me wrong, I am not trying to discredit react.js. It’s great for solving the problems that it was invented to solve effectively. The issue is react and redux concepts — presentational and container components, controlled forms, state management are applied for each and every piece of functionality, even for very simple one like Login form, which only contributes to the complexity of the code and its size and performance, but does not solve the problem more effectively. The art of keeping it simple which is the paramount quality for a professional developer gradually vanishes and I will be happy if this article or previous one makes one developer to consider alternative approach to building the complex structures with no apparent reasons for his next project. Thanks for reading!

Nick Williams on Twitter

How I imagine this code came to be: "Let's use fancy ES6 destructuring" "Seems hard to read" "Fear not! I'll leave an explanatory comment

It’s funny indeed, but at the same time it’s the typical showcase of Cargo Cult Programming — ritual inclusion of code or program structures that serve no real purpose. It might seem to be an edge case, a funky thing that couldn’t be really met in the real-world, but let’s review a piece of code from the highly rated — more than 8000 Github stars — project called Calypso.

Calypso is the new WordPress.com front-end — a beautiful redesign of the WordPress dashboard using a single-page web application, powered by the WordPress.com REST API. Calypso is built for reading, writing, and managing all of your WordPress sites in one place.

It’s clearly on the cutting edge — react, redux, webpack, babel and other 150 npm packages it depends on should allow contributors to write a modern, clean, easily readable, highly maintainable code, right? Let’s take a look. But first things first, here is a crash course on redux for those who are not familiar with it — the reducer is a pure function that takes two objects, performs some calculation and returns the new object, that’s basically it. A primer on reducer is adding a new item in the TODO list :

Multiple reducers can be composed within the single object where the keys are reducers names and the values are reducers functions. Here is the slightly modified code for one of the many Calypso reducers, extracted from here.

How long did it take for you to figure out what exactly is going on here? If you grasped it instantly then congratulations, you are true javascript ninja, because boy, oh boy, this code really gets the most out of ES6 and even ES7!

• const declarations
• computed property names
• Pulling fields from objects passed as function parameter
• Arrow functions
• Spread Object properties — it’s not even in the specification yet, but we can use it today, how cool is that!
• Shorthand Property Declarations

But to me — and I guess that I’m not alone — this piece-of-art seems too fancy, too difficult to understand, and if I were to implement the same functionality, I would do it slightly differently. This reducer updates the state of a given post publication on a given web-site, here we go:

And if you think that it’s just a matter of taste, let me show that there are more serious issues with the fancy version of code. First of all, both of the original version and the modification need to be “compiled” to javascript that is supported in old browsers(IE9 or Android Stock), the Babel REPL generates the output of the following sizes(click “full” word to review the compiled code).

• Original — full : 2012 bytes, minified : 1524 bytes
• Modified — full: 1390 bytes, minified : 1019 bytes

The difference might look minor, but in a large project, like Calypso, hundreds if not thousands lines of code written in the name of Cargo Cult might add up to the significant increase in JS scripts size which is one of the key points to avoid in order to optimize the critical rendering path.

Moreover, the modified version of a reducers works FASTER. I put together JSPerf test to demonstrate that, here is the screenshot of test run in Chrome:

Note that modified version of the code also leverages ES6 features, but it does not try to use each and every buzz-thing in the name of fanciness. In the end the code is more readable, faster and smaller. And I can optimize it even further! Let me finish this post with one of my favorite quotes about programming.

Getting and Cleaning Data

The most common approach to build a web app nowadays is bundling a bunch of npm packages — react, redux, react-redux, etc. to be used in a browser environment with a help of tools like Gulp or Grunt. These packages are often listed in the “package.json” file, so I run the following query to select public Github repositories that depend on React. It is optimized to include package.json files that locate in the root folder of a repository only.

The resulting data table contains all the run-time(isDev=0) and design-time(isDev=1) dependencies for each repository that relies on “react”:

But these are raw data — some repositories could have been removed, others might contain meaningless code and so on. To filter it out, I run the script that obtains the number of stars, forks, watchers and size of each distinct repository from the list by using the public Github API and saves the result into a CSV file:

There are two types of projects in this list :

• Auxiliary tools, libraries, boilerplates and frameworks, like zab/jumpsuit:

A powerful and efficient Javascript framework that helps you build great apps. It is the fastest way to write scalable React/Redux with the least overhead.

• Applications that rely on react and probably on these tools, like Reactive Weather(pretty cool stuff).

The target of this research is the second type — applications. They could be separated by assuming that other projects do not depend on them:

Deriving the stats

The other day I came across a tweet that made fun of JS apps

Joe Groff on Twitter

JavaScript 4K demo competition! Build an amazing app with fewer than 4096 dependencies

here is the distribution of number of dependencies among react-based ones:

75% of projects have no more than 20 dependencies and 24 devDependencies, that’s not bad at all, though the nested dependencies may add up to 4096. Here are several outliers that could achieve this number.

Top 100 run-time dependencies. It seems that redux really holds the dominant position there. It’s also interesting that jquery is still in the game and even outperforms, in terms of number of usages, material-ui package. Good job jQuery, I will never be disappointed with you.

The same for design-time dependencies:

Finally, I saw a request for good samples of applications written in React, so I compiled the list of top 1000 projects arranged by the number of stars. I sure that it’s a good material for learning the ways React is used in the real-world.

That’s all dear reader, hope that the article was interesting to read. Stay tuned!

The loop on lines 13–18 seems a little bit strange, doesn’t it? On the surface it may be unclear why the implementation isn’t as simple as something like this:

But it turns out that this “simple” polyfill may cause significant performance penalty because ECMAScript lets developers create “sparse” arrays that have arbitrary length, just like this:

The “simple” forEach polyfill called on a1, a2 or a3 array will invoke callback function 1B+1 times including 1B redundant calls for the elements of the array which do not actually exist, for example a1[0]…a1[1000*1000*1000–1] are missing and return undefined value. Not only it could mislead the callback’s code if it handles undefined values, for example counts how many of them exist in a given array, but it also may significantly increase the running time of a code. So this is completely wrong way to implement the polyfill of forEach method, moreover, you better be vigilant about using traditional for(var i=0;i<a.length;i++) loop for traversing arrays that could come up to your script from an external, potentially harmful code. Fortunately the ES5 standard explicitly specifies how forEach, map, filter, etc solve this problem:

callbackfn is called only for elements of the array which actually exist; it is not called for missing elements of the array.

The MDN polyfill gracefully implements this requirement by leveraging the distinctive feature of Array objects properties:

Array objects give special treatment to a certain class of property names. A property name P (in the form of a String value) is an array index if and only if ToString(ToUint32(P)) is equal to P and ToUint32(P) is not equal to 232−1.

In other words if an array has property with name “42” then the element with index 42 does exist in the array and vice versa. The properties are adjusted automatically when an array is being modified. Thus, in operator that is called in line 14 of the MDN polyfill checks whether an element with index k actually exists in an array. Here is the descriptive example of using this approach:

So far, so good, but here is another interesting observation : the polyfill does not check if it was called on the actual array, moreover the specification says that forEach, map, filter, etc. function:

is intentionally generic; it does not require that its this value be an Array object. Therefore it can be transferred to other kinds of objects for use as a method.

Technically, a method of one object can be “transferred” to another object by using Function.prototype.call. The most well-known application of this feature is calling Array methods on Array-like objects:

arguments object that corresponds to the parameters passed to a function is not an Array, but it has length property and indexed elements, hence Array.prototype.filter function can perfectly handle it. Note that in the code above filterFn.call returns the new Array object, so the subsequent forEach call does not require transferring.

Another example of Array-like object is HTMLCollection:

And one more trick: ECMAScript 5 introduced the way to treat the string as an array-like object, where individual characters correspond to a numerical index. You probably use this feature all the time, but take a look at how it can be leveraged to reverse a string, I guess that you probably didn’t see anything like that before:

That’s all, dear reader, for the first episode of “A Deeper Dive Into JavaScript” series. Hope you enjoyed it. Feel free to contact me if you have any questions.

Architecture

Among other mind-blowing services such as BigQuery or Natural Language API, Google provides a set of tools for super-fast information exchange and data processing.

• Pub/Sub : real-time communication service that allows to send up to 10K messages/sec, ensures encryption, guarantees delivery and operates in the Google infrastructure.
• DataFlow : real-time auto-scaling data processing service for building fast, reliable and secure Extract-Transform-Load pipelines that are integrated with with other Google services. For example a pipeline can receive the messages from Pub/Sub service, process them by using the Natural Language API and save the outcome in a BigQuery data set.

Another player in the ensemble I put together is the Twitter Streaming API that makes it possible to intercept the real-time stream of tweets that are sent from a particular location, by a specific user or contain certain phrases, for example “Hillary Clinton” or “Donald Trump”. At this point you probably figured out the whole picture as it’s quite simple, but just in case here is the sequence diagram of the architecture that performs the sentiment and syntax analysis of a stream of tweets and saves the results in a BigQuery data set. All of it, except of the Twitter back-end and client app(which is pretty simple) happens at real-time in the Google Cloud.

Implementation

I believe that the key point of this story is the incredible ease of the code that orchestrates the system shown in the picture above. The sources are available in the Github repository and you can use it under the MIT license terms. I only wrote only 150 lines of code and copy-pasted 200 more lines. For example take a look at the node.js function that connects to a twitter stream of debate-related messages and sends the tweets objects automatically serialized to JSON strings to the Pub/Sub service as soon as they are delivered. Note that the code is only interested in the tweets that are sent from the USA cities. This is done in order to make sure that the further data analysis could compare the data associated with different cities or states of the country that hosts the debate and elections. In addition it helps to prevent possible excess of the Natural Language API limits that permit only 1000 requests per 10 seconds, but the Pub/Sub and DataFlow are able to handle 10K tweets per second!

What is the “topic” argument passed in this function? In the Pub/Sub service a publisher sends messages to a topic. Subscribers components create a subscription to a topic to receive messages from it. Here is the code that creates the “debates_tweets” topic and passes it to startTracking function shown above:

The code that runs the DataFlow pipeline that subscribes to the “debate_tweets” topic, extracts a twitter text from the incoming message, processes it with the Natural Language API and saves the results in the BigQuery table is also quite simple, although it was quite an experience to write a code in Java that I didn’t touch for years.

The messages that the pipeline receives from the subscription are tweets objects serialized to JSON strings. Note that the code does not try to fully parse them and then put the fields in the separate columns of a table row, but rather saves these strings as is in the single column. The rationale behind it is the data are saved in the BigQuery that allows to include the JavaScript functions in the SQL queries and parsing JSON with JS code is a cakewalk. Moreover, the code serializes the syntax analysis results to JSON strings and preserves them in the single column — I found it pretty convenient to keep the arrays of variable lengths that way. The sentiment analysis is represented with 2 float values that can be stored in two separates columns.

Debate Night

At 9pm EST September, 26 I started the pipeline and run the twitter stream client to harvest the tweets that were posted during the first presidential debate. 2 hours later the BigQuery table has been populated with 7K rows of data that include the information about the authors of the tweets, their location and the sentiment and syntax analysis results. I just needed a good tool for data exploration, analysis, and visualization and Google offers it as well! Cloud Datalab is the browser-based tool that can be used to interactively explore, transform, analyze, and visualize data using BigQuery and Python. The amount of data I’ve collected is not that big, but still the Datalab proved itself to be extremely helpful. Here is the sample of SQL+JS query that selects the most common nouns used in the tweets and its output.

You can find the full analysis notebook here, Github nicely visualizes it. It’s just early, very basic steps though, the full analysis will take some time and it perhaps will be a subject of my next post, but the initial results are included in this story. First of all, here are some syntax analysis visualizations.

As for sentiment analysis, the Google API returns two float values for a given text : polarity and magnitude. Here is the explanation.

Here are the plots of polarity and magnitude distribution for both candidates:

It’s up to a reader to interpret these plots. What I can see here is a lot of negative feedback in the tweets about Mr. Trump.

Imagine the unimaginable

Let’s ponder for a minute on what could be possible to do by using the same technologies and even the very same code that was described in the article. This is actually the main purpose of this story, the debates stuff was used to get some attention ;)

• Startup idea : let’s say it’s Friday night in NewYork and a young hipster chooses a destination to go out. The app on his mobile phone shows the interactive map that highlights the places from where the most positive tweets are being sent. On the server side the app uses the real-time twitter analytics, just like the one that this story is about. In addition the app takes into account the analysis of photos that are shared via twitter. It is done by using the Google Vision API that provides the insight from the images, including facial expressions of emotions!
• Surveillance and Investigation. Twitter API allows to track the tweets from a specific user. By analyzing the info extracted from a stream of messages and photos that he or she posts on a daily basis it’s possible, for example, to predict(by using Google Cloud Machine Learning) an attempt to commit a crime or suicide, or just watch a person’s life. If you are a fan of “Mr. Robot” TV show, you probably recall the Eliot’s manual hack of Fernando Vera. I am pretty sure that it’s feasible to automate such a tedious task:

Fernando Vera, Shayla’s supplier. One of the worst human beings I’ve ever hacked. His password? “eatadick6969.” Aside from the massive amounts of money he spends on porn and webcams, he does all his drug transactions through emails, IMs, Twitter. The fact that the cops haven’t caught him yet is beyond me. If they had half a brain cell, they’d be able to crack his gang’s simplistic code, if it can even be called that. After only a couple of hours of timing his tweets with related news articles, I figured out that “biscuit” and “clickety” clearly referenced guns. “Food,” “sea shells” or “gas” for bullets. “rock to sleep early.” I haven’t made the direct connection to a hit yet, but the math of guns plus bullets usually adds up to one thing.

• Real-time brand monitoring is a piece of cake that would take a day or two to implement.

By the way, if you are a startup investor, private investigator or brand rep who are interested to collaborate, feel free to contact me. I will quit by boring job and we will watch the world together. Oppressive governments are not welcomed :p

Getting data

The first step in any data analysis is getting and cleaning data. In order to collect the React and Angular commit messages along with the relevant information, I used the BigQuery public Github data:

The results have been saved in the table called “react_angular_commits” that has been exported to a CSV file and downloaded for further processing that happened in R Studio. There are 7127 rows of the React messages and 8035 rows of the Angular messages.

Warming up

The first thing to compare is the length of the messages and drawing a box plot seems to be very suitable for that, here is the corresponding R code and its output:

So, the median values are close(React : 81 characters, Angular: 74 characters), but it seems that the distribution of the Angular messages lengths is significantly skewed right compared with the React’s distribution. Simply speaking it means that the Angular contributors are more “voluble” when it comes down to write a commit message. The only reasonable explanation I came with is the difference in guidelines for contributors :

• Angular enforces pretty strict format of the commit messages.
• React does not insist on any particular format.

But, I would expect the exactly opposite effect under these conditions — if everyone writes messages according to the pre-defined rules, then the length of these messages is not expected to vary a lot and vice versa. Apparently, it’s one of those cases where reality doesn’t meet expectations. Keep reading, more interesting stuff is coming :)

Analyzing word and document frequency

What are the most common words in the commits messages of Angular and React? To find it out these messages have to be split to individual words and cleaned off the stop words with the help of tydytext package.

Here is the stats of Angular’s words frequency:

Almost all of these words looks familiar to me except of “feat” and “chore”. The oxford dictionary provides the following definitions of these words:

feat — an achievement that requires great courage, skill, or strength

chore — a routine task, especially a household one

So, why are they so frequently used in the Angular’s commit messages? Let’s turn to the guidelines for contributors for explanation:

Okay, so “feat” and “chore” are just types of commits! There are other types in the top 15 words : “test”, “fix” and “docs”. Hence the next idea — the header of a commit message also includes the scope of the changes, so let’s find out the most popular scopes along with the type of changes they are affected by. The following code might seem to be pretty convoluted, but it’s not — it extracts the type and scope from the Angular commit messages and saves the result to a new dataset that is joined with the most common scopes dataset in order to filter out the rest of scopes, finally the code plots the bar chart grouped by scope.

That makes a lot of sense — for example, the most popular type of changes is “fix”, the documentation modifications affect the tutorial a lot and so on. One interesting observation : “$compile” scope demands a lot of attention in all the development areas. I don’t know a lot about $compile, but found a quite popular article that says:

View compilation in Angular is some of the most ingenious functional programming I’ve seen in JavaScript.

That explains the amount of attention required! How cool is that? Simple text mining tools available to everyone allows to reveal what’s going on in the development of one of the most popular open source libraries!

Okay then, what about React’s most common words in the commit messages?

A lot of messages have “merge”, “pull” and “request” words. In fact 2274 React’s commit messages start with “Merge pull request #xxxx” string. The culprit behind it seems to be the “Collaborating on projects using issues and pull requests” workflow and in particular the recommended approach to merge pull requests. The very same pattern pattern of commit messages can be found in other open-sources projects, for example take a look at Rails changes history. But it was not like that at all for the Angular commit messages! Why is it so? It seems that Angular maintainers use the alternative technique of integrating the contributors’ changes that boils down to applying the series of patches from the pull requests. You can find the detailed explanation in the excellent “Merge pull request Considered Harmful” blog post. The Angular’s approach along with the rules for writing the commit messages turns the changes history to a useful product story that is easy to read and analyze. Unfortunately, React commits lack this beauty, but that’s not a reason to give up. There are other text mining methods that can tell a story of React, namely term frequency–inverse document frequency can be effectively used. Here is the code that selects the words that are frequently used in React’s messages, but rarely occurred in both of React and Angular messages.

This stats is much more informative than the previous than. First of all, “spicyj”, “zpao”, “sebmarkbage”, “chenglou”, “syranide”, “jimfb”, “benjamn” refer to Github accounts of the most active contributors. These names are used in the messages like

Merge pull request #2343 from zpao/proptypes-deprecation Update PropTypes for ReactElement & ReactNode

so let’s ignore them and look at other highest tf-idf words. What about “korean” and “japanese” for example? It seems that React maintainers put a certain amount of efforts to translate the flagship documentation to CJK. That’s was not observed for Angular and its documentation seems to have the English version only. Other common words indicate the areas that are frequently affected by the changes and every React developer should be familiar with them. Funny enough, they perfectly describe almost everything you need to implement and test a React-based application, here is the illustration:

That’s all that I was able to extract from the React’s commit messages.Other methods like working with combinations of words did not help to obtain any other interesting information.

Sentiment Analysis

Finally let’s try to conduct the sentiment analysis of commit messages by using the dictionary that labels each word as “positive” or “negative”. Here is the code that plots the distribution of the most common positive and negative words in the angular’s messages:

Indeed the words labeled as “negative” are not actually negative in context! We already know that “chore” is just a type of changes suggested in the guidelines for contributors. Or, for example, “error” is certainly not negative in the “fix ‘type mismatch’ error on IE8 after each request” message. The similar picture is observed for React:

Practically it means that React and Angular contributors do not put a lot of emotion into the commit messages, for example there is only one f-bomb among 15K commit messages. See my previous post for the examples of slightly different approach to formatting the commit messages ;)

By way of conclusion

In his great book “How Google Works” Eric Schmidt explains how astonishing things are in the Internet Century:

Three powerful technology trends have converged to fundamentally shift the playing field in most industries. First, the Internet has made information free, copious, and ubiquitous — practically everything is online. Second, mobile devices and networks have made global reach and continuous connectivity widely available. And third, cloud computing has put practically infinite computing power and storage and a host of sophisticated tools and applications at everyone’s disposal, on an inexpensive, pay-as-you-go basis.

This story is the example of using these powerful technologies : the data to analyze has been obtained from the publicly available source(Google Cloud) by using the free computational power(BigQuery) that tool only 15.6 sec to process 121Gb of data , the analysis has been done in the local machine, but I could’ve leverage the kaggle kernels platform to run the code in the cloud.

• http://geeksta.net/geeklog/exploring-expressions-emotions-github-commit-messages/
• https://github.com/Dobiasd/programming-language-subreddits-and-their-choice-of-words

But the recently exposed BigQuery data provides new opportunities to look at it. Also, as I am doing it mostly for fun, practicing my writing skills and contributing to the BigQuery community which I am a big fan of, the research is pretty simple : we will look at the frequency of dropping F-bombs in the commit messages and a couple of examples of such messages.

So, “commits” table keeps the info about ~156 Million commits in the public repositories of Github, let’s select the ones that contain f-bombs in the subject or message:

Reading through these results may cause a smile or two, for example(the comments in parentheses are mine ) :

• “Add some fucking class to #a Needs more Benedict Cumberbatch” (??)
• “fix z-index, the motherfucker” (I know bro, z-index might be painful)
• “no fuck this i’m done. i am so done.”(get some coffee)
• “Rolled back license from “What the fuck you want” to the unlicense”(really?)
• “WHAT THE HELL?! Who the fuck copied the CMS file to core folder and core file to CMS folder?!”(shit happens)
• “Oh my fucking God I did a release with this debug code left in. HORROR.”(shit happens)
• “switch to java 8… fuck backward compability”(fuck indeed:)
• “Only that it needs to be the fucking opposite. Don’t drink and code…”(good idea!)
• “No fucking idea what that will do”(classic)
• “Fucking password is too fucking short”(fuck yeah)
• “Hard, or Soft? So much to do. So decent salary. “You are unhappy? You do not want to stay here? Then there are bunches of people who are waiting to take the place of you.” You stay, or you fuck off. Choose one.”

Okay, so how many f-bombs were dropped? To derive this numbers, I’ve saved the results shown above to the new “commits_with_f_bombs” table and run the next query, perhaps it’s not optimized, but it should be very easy to understand:

So, only 48K of F-bombs, not bad. Indeed if I had expanded the queries to look for the 7 dirty words and their variations, the numbers would have been much larger :)

It would be also interesting to build the model of correlation between the programming language of the repository and the number of its commits containing f-bomb, but there are 2 problems here:

• The BigQuery data do not include information about the repository language.
• The Github data actually has multiple languages per repository. For example ak72ti/Whynot which hosts 297 f-bomb commits:

Which language caused the f-commits? What if the main(DM) is OK, but the contributors struggled with javascript part? Any ideas on this one is highly appreciated.

That’s it. Have a nice commit messages :)

I code therefore I'm on Twitter

"isArray" is new "leftPad" : ~25M downloads in the last month, ONE line of code : https://github.com/juliangruber/isarray/blob/master/index.js ... #javascript #wtf #npm #github

I’ve stumbled upon “isArray” thing during the exploration of the public GitHub data available in the Google BigQuery platform. This story exposes some new interesting findings I’ve discovered.

The previous stories(1, 2) analyzed the contents of package.json files which are descriptors used for Node.js modules dependency management. Recently, I noticed some odd numbers in the Github data:

1. Number of files called “package.json” is over 8 Million.
2. Number of contents of the files called “package.json” is less than 1.5 Million.

Huh? What is going on?

The answer is pretty simple — there is a huge number of duplicate package.json contents all over the open source code hosted in GitHub. Google BigData keeps the unique contents only, based on their hash I guess. The contents table actually has “copies” column which indicates the number of duplicates, so let’s summarize the number of copies of package_json_content table that was described earlier:

Cool, the total number of package.json files is now closer to the one from package_json_files table, they are not equal though, but let’s ignore it for now and move to the next query — what is the average number of the exact copies of a package.json file?

Let’s have a look at this phenomenon closer. I’ve composed and run the following query to extract the number of duplicates for each package.json file along with its NPM name, URL in github(one URL per each duplicate) and number of copies. The query selected the records with number of copies greater than 100 in the name of simplicity.

The results have been saved in the new table called package_json_duplicates, then I executed yet another query that groups the data by the file id and sorts them by number of copies in reverse order:

Here are top 20 results:

JSON_ERROR in repo_name column means that a package.json file can’t be parsed, there are multiple reasons of that:

1. Empty or invalid file, even best of us sometimes forget to clean up before pushing the commit.
2. package.json is dynamically built on compile-time, here is the example:

But now let’s take a look at the 5th row of the results..It says that there are 47343 instances of package.json file of isArray repository..and actually there are more than that! The total number is 99692..!!WOW!!

Think about it — the module that exposes very basic code:

was used in thousands of projects all over Github, as the direct dependency or the “inherited dependency”(project B depends on project A which depends on “isarray”). Moreover, it seems that people don’t care about keeping the source code clean and commit node_modules folder of their projects carelessly, here is the snapshot of some paths to isarray’s package.json

Is it just me or it resembles cancer? A ton of duplicate , sometimes sick, code spread over all the GitHub/npm ecosystem and increased the size of the projects’ source code, like a tumor. This is why Github hosts over 8 Million package.json files and only ~1.5M of them are unique.

Okay, I got it..developers are pretty busy and don’t have time to write a line of code that checks if the input value is Array..but look, even if you are going to use that module, what on Earth makes you keep the node_modules folder in the source code? Is there any valid reason of that? Please check .gitignore file of your project and make sure that it excludes node_modules from the files to track, don’t contribute to Github Cancer development.

The descriptors of npm packages are kept in files called “package.json”. Among other things, they contain the list of the keywords that “help people discover a package as it’s listed in npm search”, for example:

So, let’s find out the trending keywords, shall we? The previous story explained how to select the contents of package.json files from the BigQuery Github data, the results were saved to “githubdataqueries:NpmStat.package_json_content” data table.

Before plunging into the query composition and results, let’s look at the amazing UDF feature of the Google BigQuery that allows developers to run a custom JavaScript code within the SQL query. A user-defined-code accepts the single data row as input and produces zero or more rows as output. Selecting the keywords from the content of a package.json file begs to be implemented in JavaScript because JSON is a subset of JavaScript and it can be used in the language naturally! Here is the self-explanatory code of the user-defined-function that emits the list of the keywords of a package following by the SQL query that uses the output of that function to rank the keywords:

Can you guess trending keywords?

Random thoughts on these results:

1. Keywords relevant to text processing : “string”, “text” and “parser” are in top 10. I can guess that the root cause of it is the JavaScript built-in String object does not expose functionality that developers use on a daily basis. Fortunately npm registry has a ton of LeftPad(wink-wink) and other useful string utils!
2. “cli” that stands for “command line interface” is in the 2nd place, closely-related “terminal” and “console” are in 9th and 11th respectively. A possible explanation is:

• CLI packages only make sense for node.js applications
• the initial node.js release supported only Linux
• CLI is at the heart of Linux & Unix systems

3. “http” and “browser” are in top 10. That’s simple : it’s all about web nowadays!