The blog of Luis Martin Gil

Mining 911 Emergency calls using Apache Hadoop

2014-01-05T00:00:00+00:00

This study started in order to practice the concepts learnt at BigData Techcon Boston, 2013. It uses Apache Hadoop and Python as tools to get the job done.

Self-moved by my internal curiosity, The goal is to understand the flow of the 911 calls for a given time-period. The 911 industry doesn’t have a huge level of call traffic but it demands a strictly well-engineered network design. Knowing how our servers are being used is a must. This study will help to understand the VoIP traffic of 911 Emergency calls.

This is @luismartingil solution. More info here.

Where am I getting the dataset from?

Dataset is taken from different random-selected SIP b2buas from our IP network. These servers route calls based on different SIP headers and databases using the Session Initiation Protocol. Obviously all the SIP messages they send/receive are properly logged with the internal state of the servers. Today I’ll take some log files from these servers and have fun with them.

Log files have lines with the following structure: {INVITE sip:number@domain:5060 SIP/2.0, SIP/2.0 100 Trying, SIP/2.0 180 Ringing, SIP/2.0 183 Session Progress, SIP/2.0 200 OK, BYE sip:domain SIP/2.0}.

What stats am I looking for? {1, 2, 3}

1) Overall days.

2) Calls for the given days of the week.

3) Calls for the given hour of the day.

Defining the problem in Hadoop

We have to properly model our solution in the Hadoop world, this is required for Hadoop to understand what we are trying to do. I consider this step as the hardest one. If we can’t model our solution into a Hadoop’s valid input we won’t be able to use it and to get the benefits of the parallelism and distribution in the cluster.

Two different methods need to be defined: map and reduce. I love drawings, let’s visualize this (high resolution here):

Main idea of my solution is to let the map ‘bucketize’ (make buckets) the input and let the reduce count them all.

a. Mapping

map gets the log files from our SIP servers as input and generates keys-value pairs. The concept is very similar to creating different hash functions. Our keys are defined differently based on the nature of {1, 2, 3}.
- a1) Overall days, we want each day to count as a key. Mapper generates : 01/03/2013 1, 02/06/2013 1, 21/01/2013 1, 01/09/2013 1, 01/11/2013 1, …
- a2) Calls for the given days of the week, each day of the week will be a key : 06sat 1, 02tue 1, 01mon 1, 00sun 1, 00sun 1, 06sat 1, …
- a3) Calls for the given hour of the day, mapper will generate key-value pairs based on the time of the day : 03am 1, 02pm 1, 02am 1, 06am 1, 04am 1, 02pm 1, 02pm 1, 11pm 1, …

Key assignation process based on buckets

b. Reducing

reduce properly joins map’s output and provides the final result. In our scenario it just merges the key-value pairs and sums the values associated to a key. In this case it’s the same for {1, 2, 3}.
- b1) Overall days, reducer generates : 01/03/2013 D1, 02/06/2013 D2, 21/01/2013 D3, 01/09/2013 D4, 01/11/2013 DN, …
- b2) Calls for the given days of the week, reducer generates : 06sat W1, 02tue W2, 01mon W3, 00sun W4, 00sun W5, 06sat WN, …
- b3) Calls for the given hour of the day, reducer generates : 03am H1, 02pm H2, 02am H3, 06am H4, 04am H5, 02pm H6, 02pm H7, 11pm H8, …

Are there more solutions to this problem?

Yes. I consider myself a learner in the Hadoop world, there might be better Hadoop implementations to this easy-problem but this is the approach I came up with. It doesn’t mean that is the only one by any means. You can have more information about the MapReduce model here.

Does Hadoop cluster look always the same?

No. In a real cluster map-reduce takes place in a multi-level hierarchy. This picture gives an idea of what it could look like in a more realistic scenario (high resolution here).

Where are your mapper/reducers?

Here you can find all the source code. Mapper and reducer are both implemented in Python. Mapper {1, 2, 3} is selected using an env variable, which makes things easy when using the BASH script to populate all the jobs with one command.

Doing the job. Getting the results

I’m using Cloudera quickstart Virtual Machine 4.4.0-1 with Hadoop 2.0.0-cdh4.4.0 for the cluster simulation. You can have a quick look to the commands used here. Basically I copied the files to be mined into the HDFS file system and executed the run.sh. Results were copied back from the HDFS to the VM by the script. Here is an output of one of the jobs. As you can see the reducing process starts before the mapper is done.

14/02/01 23:49:17 INFO streaming.StreamJob:  map 0%  reduce 0%
14/02/01 23:49:32 INFO streaming.StreamJob:  map 3%  reduce 0%
(...)
14/02/01 23:53:06 INFO streaming.StreamJob:  map 93%  reduce 15%
(...)
14/02/01 23:53:24 INFO streaming.StreamJob:  map 100%  reduce 50%
14/02/01 23:53:34 INFO streaming.StreamJob:  map 100%  reduce 100%

1) Overall days
- Last months each b2bua instance has been getting an aprox average of 6500 calls per day. So some groups of counties of Indiana have been getting around 4 calls per minute. This number could sound ridiculous from the carriers’ perspective, in the 911 world is a valid number though.
- There is a relevant peak in the plot. On November 17th 2013 each instance got around 12600 calls, which almost doubled the normal traffic of the servers. What did happened that day? We had one of the worst weather conditions of the last years. Tornados were all over the midwest of the USA, hitting hard some counties in Indiana. Tornado outbreak. I’m glad we did help the people. Very interesting.

2) Calls for the given days of the week (From September 1st 2013 - Jan 1st 2014)
- Social interactions are pretty defined in the next plot. Weekends are pretty much the hottest points in the 911 call-generation for a given week. That’s probably what all of us guessed, but it’s always nice to backup the thoughts with the real-data.

3) Calls for the given hour of the day (From September 1st 2013 - Jan 1st 2014)
- Next plot also makes a lot of sense. Being 5pm the rush hour and people usually sleeping during night (funny), this sinusoidal wave perfectly defines the normal schedule for the people interactions (and 911 call-generation).

( Graphs are made using Gnuplot and tweaked with Gimp. Open source powered! )

Conclusions

People’s habits are described in these plots, also events like the tornado outbreak are pointed in our data. Results are reasonable and pretty logical. Using technology to approach these studies is pretty neat. Learnt several things in the mining side as well. Very interesting project overall.
I would like to thanks INdigital, my actual employer. I’m proud to be part of a team of devs that are making the 911 Emergency technology better. We are doing a job that has a great impact in the society making it better for the people to help each other.

Did I really need to use Hadoop for this?

For files of this size of this configuration, obviously no. I can easily do this using a terminal command piping the map and reduce myself. We start to get the benefit of Hadoop using a real cluster and mining X files Y times bigger (X, Y will be something interesting to research). This is an enjoyable approach to do the job as the same time we play with the basic technologies and concepts of big data. Obviously this could be a good starting point as a research for a valid scalable solution.

Getting a fair result with an unfair coin

2013-11-20T00:00:00+00:00

How can you get a fair coin toss if someone hands you a coin that is weighted to come up heads more often than tails?

Solution

This is @luismartingil solution. Please check all the Python source code here. This solution uses matplotlib for the plots.

We can have two different types of results when throwing the coin: {heads, tails}
We are going to have two players playing the game: {player-heads, player-tails}
player-heads is the player that chooses heads. player-tails the one choosing tails.
A fair game means that both players have the same probability of winning (50%).

We first need to know how unfair the game is in order to make it fair. We have to know how many times the coin comes up heads vs the times it comes up tails. We do this throwing the coin N times, where N is a huge number. After doing this we have a percentage of the times the coin comes up heads, lets call it p1. We can define the percentage of times the coin comes up tails as p2.

Based on the problem: - p1 and p2 are in this range [0, 100] - p1 will be bigger than p2. p1 > p2

We can assume that p2 = 100 - p1.

In order the game to be fair, player-heads has to win more than the p1 times of the times played in order to win the game. The bigger N is, the fairer the game is.

Proving the solution

Always back what you are saying!. A neat way to back my solution is to write some code and perform a simulation. That’s what I did.

Game simulation

 1 class Game(object):
 2     coin = None
 3     simulation = None
 4     def __init__(self, c):
 5         self.coin = c
 6     def simulate(self, n):
 7         self.simulation = {HEADS : 0, TAILS : 0}
 8         for i in range(n):
 9             self.simulation[self.coin.throw()] += 1
10         return self.whoWon()
11     def whoWon(self):
12         if self.simulation:
13             # PLAY_TAILS wins in case of even
14             return self.simulation, PLAY_HEADS if self.getCondHeads() else PLAY_TAILS

UnfairGame

Original game. It doesn’t know if the coin is weighted or not.

1 class UnfairGame(Game):
2     def getCondHeads(self):
3         return self.simulation[HEADS] > self.simulation[TAILS]

FairGame

A handicap is applied to the player-tails. Takes into account thw weight of the coin (coin.getPercent()).

1 class FairGame(Game):
2     def getCondHeads(self):
3         # PLAY_HEADS needs to win more to win the game. Handicap ;-)
4         # @luismartingil solution to the problem.
5         total = self.simulation[HEADS] + self.simulation[TAILS]
6         return (self.simulation[HEADS] > (self.coin.getPercent() * total))

Coins

Both FairGame and UnfairGame will be played with different coins.

Coin 1) 50% - 50%

Coin 2) 60% - 40%

Coin 3) 80% - 20%

Some plots. Playing the games

Plots-1, Player heads wins

These plots contains information about the wins of the player-head in a percentage based. Whether we have a fair or unfair {game, coin} the goal for a fair game is to see the points right in the 50% horizontal line, which would be the fairest game possible.

Some comments,

Coin 1). It doen’t matter which game we play using a fair coin (50% - 50%). It is always fair.
Coin 2-3). UnfairGame is always a looser game for player-tails using unweighted coin. This means that player-tails is likely to win in the beginning, but if he plays a bunch of times he will loose.
Coin 2-3). The more unweight is the coin, the faster it will converge.
Coin 2-3). Our FairGame solution works, it converges to 50% in all the cases.

Plots-2, Fairness ratio : How fair is this solution?

I was curious about the fact of grading a fair solution. I came up with a function which returns how fair the solution is based on how close it is from the 50% percent. You can see the code and some plots below.

 1 def processFairness(percent):
 2     """ Return a value in [0, 1] defining how fair is the percent.
 3     0, worst.
 4     1, best.
 5     
 6     Applies the fairness function based on:
 7     if x == 50 , y = 1
 8     if 0 <= x < 50, y = x/50.0
 9     if 50 < x <= 100, y=-x/50.0 + 2
10     
11     |
12     |      (50,1)
13     |         _
14     |        /|\
15     |       / | \
16     |      /  |  \
17     |     /   |   \
18     |    /    |    \
19     |   /     |     \
20     |  /      |      \
21     | /       |       \
22     |/        |        \
23     ---------------------------------------------------------------
24     (0,0)   (50,0)   (100,0)
25     
26     """
27     if (percent == 50): ret = 1.0
28     elif (0 <= percent < 50): ret = float(percent) / 50.0
29     elif (50 < percent <= 100): ret = (float(-percent) / 50.0) + 2.0
30     else: raise percentOutOfRange('Error calculating fairness. percent:%s' % percent)
31     return ret

Some comments,

Coin 1). Coin (50% - 50%) has the best GPA and always converges to grade A.
Coin 2-3). UnfairGame always converges to rate 0 when using an unfair coin.
Coin 2-3). Our FairGame solution rates very good for a med-high N. Nothing that we didn’t know.

Conclusion

Bet your money in heads when the coin is weighted to come up heads more often than tails. Save your money in all other cases. Better than a casino.