Information is quickly piling up all over the place, and I agree with the article that the people who are able to capitalize on this are the ones that will get the big payoff. I especially like the idea of calling these start-ups “wildcats”, that perfectly captures the wild west atmosphere that is going to start happening.

The neat thing is that a lot of this information is out there for free, the real value is how people are going to aggregate those individual data streams into a new and often unexpected products. Take twitter for example (are you following me on twitter?), it is a conduit to what is going on in the hive mind of the internet. This site seems to be gathering up the trends on twitter and then adding news articles about some of the things that are hot.

That is pretty neat: Data is generated in the form of people tweeting about Topic X, as X becomes more “important” (in this case more people discuss it so that it rises above other topics) it gets published to the “trending” list. This website then goes in and looks at that list adds more data to the conversation by reporting news about topic X. That way the separate data points are tied together to show that there is a relationship between them, and in the process makes the data more valuable to the end users (by supplying more context, etc.)

Big data is going to lead to a lot of big opportunities. All we have to do is find the data, combine it in the right way, and perform the right data analysis on it. And unlike bit oil, big data is going to be around a very long time.

What you can measure, you can manage.

I just watched a new video from Giles where he talks about how you can improve your programming productivity. Its a really good short video that hits the nail on the head. If you want to make a change, guessing about what to fix won’t cut it. You need to measure what you are doing, and then adjust accordingly.

The quote at the top was in the video, and I have to say I don’t think I had ever heard it before. But as soon as I heard it I knew it was 100% correct. Think about it: everyone who is in charge of things in you life (bosses, teachers, parents, etc.) track what you do to some extent. When you step out of line, they know about it and are able to let you know. Why? Because they are “measuring” you progress via grades, work done, chores completed, etc.

So if it works for them, why can’t it work for you? Track the things that are important to you, and see if you can make a positive change.

I’ve been hemming and hawing lately about tracking things like my programming projects or seeing if my neighborhood association really is increasing the house values. The time for action is now.

My first step: Putting widget on this blog to track my BitBucket RSS feed. If I’m going to work on a project out in the open why not let everyone know about it? This way if I’m not being productive, it will be pretty visible.

(As a side note, this is something that Giles has mentioned before that I really believe in: If you are a programmer, you should have some project out in the public eye. Open source is a good thing. Contributing to open source is a great thing. Being known as a programmer who contributes to open source software is the best thing.)

My next step: Start treating time tracking on my projects as a first class citizen. I’m starting a new sprint tomorrow: I’m going to track my time better. Also, I’m going to add some tasks to my Satellite Tracking project so I can make sure I’m on task when I’m working on it.

Thanks Giles, that video was a good kick in the ass.

I’ve written a Google App Engine application to try and help people figure that out. Paste in a copy of your resume and a copy of the job description, and it will try and figure out how well of a match you would be for that job.

Check it out: http://app.ironboundsoftware.com

I’m really impressed with the Google App Engine environment (go Python!) and had fun writing this. Hopefully this will help people out in their job hunt. Times are tough, and hopefully this little application will help someone get into the perfect job for them.

Try it out and let me know what you think!

Overall it seems like a good approach to aggregate this data together to get the “big picture” of where one’s website stands in the web. I’ve tried this but I’ve noticed that Technorati doesn’t seem to report the numbers I expect it would.

For example: I got a link to my blog from Hip Egg a few weeks ago, and this link has not been reported on Technorati. I know this blog is small potatoes in the grand scope of the universe, but it strikes me as odd that my more recent updates are featured in my “favorites” (as is Hip Egg’s posts), yet the link from him hasn’t shown up.

My working assumption is that Technorati has some kind of filter where lower popularity sites aren’t “updated” as often as the bigger sites. Either that or the link database is broken. But links for other sites seem to be working, although I’m not watching them as closely as I watch my own stats.

At any rate I wonder how many other blogs (or sites in general) are suffering from this problem. Metrics for websites are difficult to agree on, so a site or sites where a reputation can be established and measured becomes more and more important. Aggregating data from multiple sites is a good start, but if there are too many “issues” with how a site is ranked, then the data becomes suspect and it becomes harder to get a clear picture of what’s going on.

Graphs, and what you can do with them, never ceases to amaze me. The article talks about how looking at the relationships between users on eBay can help uncover fraud and the accomplices that help keep it going. They do this by seeing if the relationship between groups of users turns into a biparte graph. That is, there is a concentration of links between two groups of users, and few links with other users.

The links between users refers to the “feedback scores” that eBay users use to determine a user’s trustworthiness. Usually, when someone begins ripping others off, they get bad feedback until it reaches a point where no one will do business with them. This is the way that community sites commonly work (i.e. letting the population of users determine each other’s rankings). The issue is that sometimes there are people lurking in the shadows assisting the fraudster, but because they are never the target of the bad feedback, they are able to keep going supporting new fraudsters (i.e. a new user id).

This setup allows a scammer to setup a new user id and get its feedback levels boosted quickly without having to engage in a lot of “legitimate” transactions. Think of it as passing a baton in a relay race: Instead of one person running a mile, why not let several people sprint as fast as they can for a quarter-mile and then hand off to someone “fresh”.

Think about it: When was the last time you checked someone’s feedback ratings on eBay? Probably right before your last purchase/bid. But, when was the last time you checked the people who gave the feedback to see what their reputation was? Probably never. I know I had never thought of this before reading the article.

This technique produces a graphical representation of this relationship which stands out quickly to a user. (Also, as side note, there are mathematical formulas that would/should see this also.) A normal user would probably have a relationship graph (over 2 or more degrees) that looks like a star-burst pattern. A potential scammer would show up in a clustered bipartite graph. This would give the community of users (eBay bidders) a powerful tool to determine who is the real deal and who is trying to give them a wooden nickel. Very cool stuff. Graph theory to the rescue!

I’ve talked with Hip Egg and Jym Khana about stocks before and one topic that I bring up every now and then are penny stocks. As I’m sure most people with an email account know, there’s a ton of stock related spam going around these days. Most of it appears to be the pump-and-dump variety in which the scammers hope that people will purchase the suggested stock causing the price to rise so that they can sell their shares (that they purchased before sending out the email) at an inflated price. This technique has been around for ever, but it seems to the flavor of the month for scam and con artists.

The main questions that we usually talk about are a)Does any one actually get rich doing this? and b)Just how “influence-able” are these low priced stocks? Well, today has been a banner day for answers, I came across two articles talking about the scams:

• Stock Spam Effectiveness Monitor (via Joel)
• Spammers as scammers
• Stock Touts and Corresponding Market Activity

While reading these I saw the simplest idea yet to help stop those spams: Simply watch the stocks and see who bought a lot of stock before the email was sent, and who sold a lot right around the sell date in the email.

That idea is pure genius. It targets a potentially large group of people, but the probabilities are that a pattern will emerge that a small group of people are moving from one stock to another. At a minimum those groups would be a starting point for a fraud investigation. More than likely, those would be the people responsible for sending out the emails. And since the spammers are kind enough to send these messages to just about everyone on the planet, it shouldn’t take too long to gather a nice body of evidence (or actionable intelligence). From what I understand in the past scams like this have been hard to track because the scammers can move quickly. But now that they are announcing their moves in advance, it should be pretty easy to set up a system to monitor spams, then watch the stock activity… It just seems so simple, that it should work like a champ!

Traffic waves are what happens when there is a slow down for some reason in the flow of traffic. The cause could be an accident, a glare from the sun that blinds people, or just about anything that causes traffic to slow down. As people slow down, the drivers behind them also have to slow down. As the first drivers pass the “distraction” that caused them to slow down, they begin to speed up. But this speed up does not get propagated to the other drivers right away, so the drivers further back in the pack are still going slow (and thus causing the drivers behind them to slow down). The result is a “standing wave” where the cars slow down. As long as there are more cars heading to wave, the wave will persist (assuming the original distraction is gone), once the rate of cars coming towards the wave slows, the wave breaks down and disappears.

Anyways… The article has an interesting idea of using Police patrol cars to help break up the wave by having them in the traffic (several miles before the slowdown) driving at a “slower” speed than the normal traffic flow. Because people are not likely to drive fast past a cop, this effectively slows the rate of cars flowing into the wave, which helps to break it up. Its a really interesting idea, and I think it could really work.

One thing that I’m not so sure about is the authors assertion that he could affect the same thing by driving at a steady rate (i.e. avoiding stop-and-go and trying to keep a good buffer distance between himself and the car in front of him). His idea is that once he does this it encourages the drivers behind him to do the same thing. I’m not sure I agree with this, I see a lot of impatient drivers on a daily basis. Maybe its just here in Atlanta, but if there’s a half a car length in front of you, and your lane is moving, someone is going to try and get in there.

Having said that, I do like the spirit of the idea and I’m going to try it out next week as I drive in traffic. Who knows, maybe a few other people will read that article and try the same thing. Anything that keeps the traffic moving is a good thing in my book.

Today I saw this analysis on the Netflix forums:
Netflix Prize: Forum / Miss Congeniality. In this posting there is a breakdown of the movie ratings, and it explains a few things. Like the most loved or the most hated. Its a pretty interesting read (especially if you are feeling lazy and don’t want to do the SQL, which is the category I fall into).

Finally, a mashup that does something useful.   As the article points out there are certain patterns of usage that probably would raise red flags. What surprises me the most though is people aren’t excited about this type of technology being used.

Given the large size of the user base of MySpace, it is pretty impractical to have a set of eyes on every user and every posting that goes on. Granted, there is a real possibility of false hits when doing an automated search like this. But given that a computer can sift through the set of millions of possible hits and narrow it down to a few hundred (which can then be followed up by a human), to me it is a no brainer.

The best part of doing an automated data mining scan of a site like MySpace would be that its a computer, not a person doing the scanning. Computers don’t make judgements, or laugh at your music choices, they just scan. To me it seems like this would be the best of all possible worlds: Let the computer make the rough pass over the site, and pick out the most “questionable” users/postings for human followup. That way the “invasion of privacy” (if there even is such a thing on the internet anymore) is limited. And by making sure that a human is doing the follow up, we hopefully remove the problem of the over-zealous filter that assumes everyone is bad.
I’m curious to see what the reaction is when the reporter releases the code that lead to the investigations in the story. Will MySpace adopt its usage? Will vigilante surveillance groups pop up and patrol the internet? Interesting times lie ahead…