Information is quickly piling up all over the place, and I agree with the article that the people who are able to capitalize on this are the ones that will get the big payoff. I especially like the idea of calling these start-ups “wildcats”, that perfectly captures the wild west atmosphere that is going to start happening.

The neat thing is that a lot of this information is out there for free, the real value is how people are going to aggregate those individual data streams into a new and often unexpected products. Take twitter for example (are you following me on twitter?), it is a conduit to what is going on in the hive mind of the internet. This site seems to be gathering up the trends on twitter and then adding news articles about some of the things that are hot.

That is pretty neat: Data is generated in the form of people tweeting about Topic X, as X becomes more “important” (in this case more people discuss it so that it rises above other topics) it gets published to the “trending” list. This website then goes in and looks at that list adds more data to the conversation by reporting news about topic X. That way the separate data points are tied together to show that there is a relationship between them, and in the process makes the data more valuable to the end users (by supplying more context, etc.)

Big data is going to lead to a lot of big opportunities. All we have to do is find the data, combine it in the right way, and perform the right data analysis on it. And unlike bit oil, big data is going to be around a very long time.

I own all 10 seasons on DVD. Even the last one, which *really* wasn’t the greatest.

When Stargate Universe was announced I was skeptical. I wasn’t big on Stargate Atlantis, but I decided that the new series deserved a fair shake.

So I watched the first half of the season. It went from “Not bad” to “Ehhh” to “*sigh*” to “Why bother” pretty quickly for me. Since it was taking a 4 month break between the first and second half combined with me not really caring about the major characters, I never watched the second half.

Today on the most excellent Stargate site, Gateworld.net, they released some ratings numbers. It looks like I wasn’t alone in my decision to tune out. Check out this nifty plot I made of the ratings over time:

*sigh*…..

This reminded me of something I heard about back when the iPhone was just getting started: The German Tank Problem.

Its basically a way of statistically estimating the maximum number of “items” that have been produced based on the serial numbers of the item. This method works when the serial numbers are in sequential order, and basically allows you to produce a somewhat realistic estimation (as opposed to a wild-ass guess).

Estimation is one of those skills that most people (myself included) could stand to improve a little bit. In the wikipedia link above, this is shown by the intelligence estimates the Allies had for the number of tanks the Germans were producing in the middle of WWII. The initial estimates were really high (1,000 per month), but using statistics based on the serial numbers of crankcases from captured or destroyed German tanks showed that the number might be lower (around 200 per month, or 1/5th of the original. After the war when the factory records were looked at, the true number was a lot closer to 200 than 1,000.

Math FTW

The formula is pretty easy:

N=m + (m/k) -1

Where m is the largest serial number observed, and k is the number of serial numbers seen. The variance is roughly equal to:

(N^2)/(k^2)

Which means that the standard deviation is roughly:

N/k

So, how does this apply to the iPad’s initial orders? Based on the data points of 2 known orders spaced 50,000 numbers apart (keeping in mind these order numbers also probably included orders for items other than iPads). So plugging those two numbers into the equation we get N=

>>> m=50000
>>> k=2
>>> N = m + (m/k)-1
>>> N
74999
>>>

Wow, almost 75,000… That seems like a really big number. The question we should then ask ourselves is “How realistic is this number?” Using the standard deviation and variance we could find out how spread out our numbers are:

>>> (N**2)/(k**2)
1406212500L
>>> N/k
37499
>>>

Wow. Those numbers huge. And that is a very bad thing. The bigger the standard deviation and variance, the more less accurate the estimation. Another way to approach this analysis is to look at the confidence interval and see how big it is. The wikipedia article has a handy formula for finding the confidence interval which leads us to the estimation that there are (based on k=2 and m=50,000) between 50,000 and 225,000 iPads ordered!

Those numbers are lying

Why? There’s two reasons: The main one is that our sample size (of 2 orders) is waaaaaaaay too small. Its like trying to guess how big your grocery bill is by averaging the price of two items, and then multiplying it times the number of things you bought. That estimation will be way off.

As an example, if there were 20 orders to work with (k=20), the high end of the confidence interval would shrink to 58,000. But that leads to the second reason why we can’t trust these numbers:

We don’t know the lower bound.

In other words, yes, there could have been 50,000 orders between the first and last data points. But what if only half of them were for iPads? That would mean that m is actually 25,000 which would drastically skew the numbers down. Remember that N that was almost 75,000? With m at 25,000 (and keeping k = 2) N drops to 37,499 which is half the original estimate!

So how many were sold?

That’s a really good question. Knowing Apple and how people love their products, I bet they sold a TON of iPads. But based off these rough numbers we see in the news, we can’t really draw a good conclusion. We can get a couple of estimates which are better than nothing, but they are so numerically shaky (huge standard deviation and enormously questionable confidence interval) that they strain believability. More data points will help establish a upper bound (i.e. the estimated maximum number sold), but without a lower bound to keep us grounded the numbers will still look really huge.

One of the things I have learned just today is that loops are actually allowed in Lisp. All of the recursive thinking I’ve been doing lately has made me think that there’s always a way too write a loop in a recursive manner. But, I was looking at Neville’s Method the other day and it struck me that with it’s nested loops, that could be quite a challenge to write it in a recursive manner. While trying to find an example of nested loop in Lisp, I found all kinds of cool links I thought I would share:

• Lisp style tips for the beginner
• Successful Lisp
• CLMath

Go check them out, they are great resources!

Graphs, and what you can do with them, never ceases to amaze me. The article talks about how looking at the relationships between users on eBay can help uncover fraud and the accomplices that help keep it going. They do this by seeing if the relationship between groups of users turns into a biparte graph. That is, there is a concentration of links between two groups of users, and few links with other users.

The links between users refers to the “feedback scores” that eBay users use to determine a user’s trustworthiness. Usually, when someone begins ripping others off, they get bad feedback until it reaches a point where no one will do business with them. This is the way that community sites commonly work (i.e. letting the population of users determine each other’s rankings). The issue is that sometimes there are people lurking in the shadows assisting the fraudster, but because they are never the target of the bad feedback, they are able to keep going supporting new fraudsters (i.e. a new user id).

This setup allows a scammer to setup a new user id and get its feedback levels boosted quickly without having to engage in a lot of “legitimate” transactions. Think of it as passing a baton in a relay race: Instead of one person running a mile, why not let several people sprint as fast as they can for a quarter-mile and then hand off to someone “fresh”.

Think about it: When was the last time you checked someone’s feedback ratings on eBay? Probably right before your last purchase/bid. But, when was the last time you checked the people who gave the feedback to see what their reputation was? Probably never. I know I had never thought of this before reading the article.

This technique produces a graphical representation of this relationship which stands out quickly to a user. (Also, as side note, there are mathematical formulas that would/should see this also.) A normal user would probably have a relationship graph (over 2 or more degrees) that looks like a star-burst pattern. A potential scammer would show up in a clustered bipartite graph. This would give the community of users (eBay bidders) a powerful tool to determine who is the real deal and who is trying to give them a wooden nickel. Very cool stuff. Graph theory to the rescue!

I’ve talked with Hip Egg and Jym Khana about stocks before and one topic that I bring up every now and then are penny stocks. As I’m sure most people with an email account know, there’s a ton of stock related spam going around these days. Most of it appears to be the pump-and-dump variety in which the scammers hope that people will purchase the suggested stock causing the price to rise so that they can sell their shares (that they purchased before sending out the email) at an inflated price. This technique has been around for ever, but it seems to the flavor of the month for scam and con artists.

The main questions that we usually talk about are a)Does any one actually get rich doing this? and b)Just how “influence-able” are these low priced stocks? Well, today has been a banner day for answers, I came across two articles talking about the scams:

• Stock Spam Effectiveness Monitor (via Joel)
• Spammers as scammers
• Stock Touts and Corresponding Market Activity

While reading these I saw the simplest idea yet to help stop those spams: Simply watch the stocks and see who bought a lot of stock before the email was sent, and who sold a lot right around the sell date in the email.

That idea is pure genius. It targets a potentially large group of people, but the probabilities are that a pattern will emerge that a small group of people are moving from one stock to another. At a minimum those groups would be a starting point for a fraud investigation. More than likely, those would be the people responsible for sending out the emails. And since the spammers are kind enough to send these messages to just about everyone on the planet, it shouldn’t take too long to gather a nice body of evidence (or actionable intelligence). From what I understand in the past scams like this have been hard to track because the scammers can move quickly. But now that they are announcing their moves in advance, it should be pretty easy to set up a system to monitor spams, then watch the stock activity… It just seems so simple, that it should work like a champ!

So how do you prevent this?

For me the secret has been to set up two new play lists. One is dedicated to new songs (i.e. just bought or ripped), and the other is for songs that haven’t been played a while.

With iTunes, you can setup a play list that will select songs based on certain fields. For me, most of my play lists revolve around the “rating” of the song. 1 Star means I don’t like it, 5 Stars means its the best thing I’ve ever heard. As a consequence, I have a lot of songs that fall into the 3 to 4 star range. Randomly choosing songs from this pool is ok, but for some reason I always seem to wind up with the same core groups of songs, and like I said earlier, they are starting to get stale.

It turns out that there is another field that iTunes keeps track of, the “last time played”. This is interesting because now we can build a play list based not only on how much we like the song, but also how long it has been since we have heard it. Combining the two ideas together leads to an interesting new play list. Here’s a picture of how I have mine setup:

The real beauty of this play list is that as songs are listened to, their “last played” date is set to now, and when I sync up the next time, a new song will take its place. This way, I can keep listening to the songs I like, but don’t have to worry about stale songs because the play list is always being refreshed.

And as I listen to songs on the Mac, this updates the last played dates also, so the net effect is I’m adding a lot of chaotic variability to the play list. Which in turn means that the songs on the play list will tend to be more “random” because there are two sources of input (the iPod and iTunes) that are influencing the results of what gets picked.

Today I saw this analysis on the Netflix forums:
Netflix Prize: Forum / Miss Congeniality. In this posting there is a breakdown of the movie ratings, and it explains a few things. Like the most loved or the most hated. Its a pretty interesting read (especially if you are feeling lazy and don’t want to do the SQL, which is the category I fall into).

Think about it: They are giving away money in exchange for people shopping at their stores. But, they aren’t going to loose money, and if anything they will gain huge insights into the consumer base in an area!

For example, lets say they mail out 10,000 of the cards. And lets say there are only two values for the cards, $1, and $10,000. The smart thing would be to send out as few of the $10k cards also possible, so in this exercise we’ll assume 1 was sent out. That means that the other 9,999 cards were all worth $1 which means the total value of the “prize” is $19,999. That is, assuming that all of the cards are used (which in real life probably wouldn’t happen for a variety of reasons), and ignoring other things like the cost of the mailings, the time of the staff to prepare things, etc.

A lot of people who get these cards are probably like me, they have a bunch of small projects around the house, but they just don’t have any motivation to do any work on them (because they need parts, they need time, etc., etc., etc.) This card comes along and they decide “Hey, I could win $10k, why not go down there and get some of the supplies I need anyway, and then I can find out what this is worth!”.

So lets assume that 50% of the people who get the cards decide to use them. My wife and I wound up spending $75, but I’m sure that other people will spend more, and other will spend less. For this little exercise, lets figure on the average amount spend is $50 per person. That works out to $250,000 spent at the store! Even if one of the cards happened to be the winning $10k card, Home Depot will still have brought in over $200k of business. If the average spent per customer was higher and/or more people used the cards, the amount would be even higher.

So, for a $20k investment, the business got back $250k in sales. For a company so big, that’s probably a drop in the bucket, but the effect is something that can’t be ignored. And that’s not even the good part!

For those of you who’ve never seen one of these gift cards, they are basically a plastic card with a bar code on the back. The bar code is scanned at the register which queries the main database to find out the value of the card. Pretty standard stuff. But then it occurred to me that I thought the card was addressed by name to me and my wife. I could be wrong, it might have said resident, but at any rate the ZIP code was on the address.

What this means is that there is now (potentially) a connection between how much was spent at the store and what ZIP code the customer lives in. This is a gold mine of data, it tells you so much about your customer base. Off the top of my head I could think of the following:

Continuing with the example above, imagine if the results of this campaign showed that there was a higher turn out from certain part of town those parts could be investigated to find out what kind of homes are there. It could be that the homes in that area are older and thus more likely to be getting “fixed up”. Likewise, if the turnout was low in another area, research could turn up that the houses are newer and less likely to have a need for large amount of work (i.e. the owners are less likely to spend higher amounts of money).

There’s a ton more that could be derived, but the important thing is that the lessons learned can be applied to the next marketing campaign. There’s all kinds of targets that can be aimed for: a higher turn out, a larger purchase per customer, moving a certain type of product (like paint, etc.), the list is endless.

If nothing else, the whole thing has shown that I’m willing to put a lot of thought into something if it will keep me from having to do home repairs.

I’m using randint() to simulate dice so I’m curious to see if the number distribution is even across the numbers 1 through 6. Below is my test code:

for x in range(6):
    counts = [0,0,0,0,0,0,0,0]
    for x in range(ONE_MILLION):
        counts[d6()] += 1
    for i in counts:
        print i, ',',
    print ''

Each time d6() (my wrapper around randint()) is called, it returns a number 1-6. This is used as a look up into the counts list, and the number there is incremented by one. I have 0′s on both sides of the 1-6 slots just to make sure it really is returning a correctly bounded value. The numbers in each row should sum up to 1 million.

By running this 6 times, I should get an idea of where the numbers are falling to make sure there is an even distribution. (Truly random numbers will have an average distribution over the long term, if they are grouping around one number, then they random number generator is not doing a good job.) I took the total of each column (which should be very close to 1 million) and then found the percent error ( ((amount – expected) / expected) *100) (omitting the absolute values that are usually used). The average of the percent errors was 0. This leads me to believe that the distribution of random numbers generated by the randint() function are sufficiently random for my uses.

Now that I have stated this, I have no more excuses but to continue on with coding the game that will use said function in a dice throwing function. Below is the spreadsheet of my data as generated by Google Spreadsheets.

0 166367 166368 166846 166996 167006 166417 0 1000000 0 165463 166853 166669 166644 167031 167340 0 1000000 0 167284 166470 167052 166227 166123 166844 0 1000000 0 166893 166893 165958 166655 167011 166590 0 1000000 0 166887 166370 166124 166672 167160 166787 0 1000000 0 166802 167174 166724 165704 166800 166796 0 1000000 Total: 999696 1000128 999373 998898 1001131 1000774 % Err: -0.304 0.128 -0.627 -1.102 1.131 0.774 Avg % Err: -0

By the way, this data was generated with python 2.4.3.