-Nick

i’m talking about a more general purpose application that takes minimal hints about the context of your raw information. in other words, it knows how to deal with your input better than you do yourself.

it has to be lite too. i cannot stress this more. i hate loading Adobe Photoshop whenever i need to change a BMP to JPG. ACDSee makes this faster, but still takes helluva long time loading / registering plugins. we need some quick and easy fix for all our analytical needs, like the smart nerd-friend who always does your hw.

imagine for instance, cutting-n-pasting a table in ASCII that is probably row-major or column major, who knows. you want to interpret this data because you are a bio-med researcher investigating correlation between cancer and nutrasweet, because you are an accountant trying to help your boss with tax evasion, or simply because you are too cheap to buy Quicken to finance your kid’s education. the possibilities are endless. i can name an infinite number of situations where we have all this data in our hands, and don’t know what to do.

enter the data-miner: you paste it in, and voila, it interprets the data from every angle of attack. if you give hints, it does something more specific. if you don’t, u get an organized, succinct summary of what you’re looking at, in charts and graphs. the analysis will include all the basics like stddev, mean, variance, covariance, correlation, then maybe more advanced stuff like skew, chi-square, network neutrality, clustering, LDA, PCA..

the tricky part is the UI. how do you present charts and graphs without exposing the user to ugly parameters, or rudimentary scripting. it seems most math packages nowadays such as WEKA, R, Mathematica, Matlab, more or less depend on some verbose mathematical hieroglyphics most people don’t understand..

i think, either these ppl have to go read a book, or the programs have to get smarter. with technology always following a gradient-descent down the lazy curve, don’t expect the former. along the lines of your G.T.D. philosophy, there’s still room for the switchblade of data-mining. u just need to find the sweet spot between form and function. it has to be 1) useful 2) simple 3) sexy.

Good call about the spreadsheet program, I think the world needs something like that. I keep seeing these Ajax based programs popping up and I’ve seen one or two spreadsheet programs, but they are not quite there 100%. As soon as they are able to read and write MS Excel files they will have a very cool program that could be worked into what you were describing. Web-based, organized and mine-able data. That is very cool. I especially agree with your “the ease of operating a post-it pad; responsive as MS Paint and as ubiquitous as Sudoku and Minesweeper” comments, I think that’s they way all software should be. I’m trying to make the stuff I write that way.

By the way, yes, I did get your last post. I’ve looked at it real quick but haven’t had a chance to read any of it. I’m going to bookmark it for later because it looks like some great reference material. Thanks for sharing and good luck with your research!

-Nick

what can you do with a table? you might apply some transformation to the row vectors. there should be a pipeline of data operations that is available to you. you can go with prefab, or you can bake your own. this should all work like Adobe photoshop layer transforms, or premier filters. everything is visual. you drag a matrix into a table, boom! it knows what you mean.

i can go on, but i got more research to do on reeb functions, shape homotopy, and graph isomorphism. good times.

it would be cool if i had a small 600 kb utility that makes spreadsheets intuitive to operate for the ordinary guy who doesn’t do his own accounting (ie. clueless with excel). it would have automatic data-mining functions built in, like PCA, LDA, and KSOM. the interface would be minimalist like a doorknob but user-friendly like an iPod.

tables are merged, sorted, split all with a mouse. the interface would all be drag-n-drop WYSIWYG. no guess-work as to what everything means. i don’t like fancy buttons and i hate clutter.

as for features, the front end would be something as powerful as VTK, giving you a variety of visualizations to choose from. there would be a simple SQL query service built in, just for the plugin developers.

not sure if there’s a niche here, but to be of any value the program must be as petite, basic, and responsive as MS Paint and as ubiquitous as Sudoku and Minesweeper