February 17, 2009
Over the last couple of years, I’ve been invested in building a distributed problem solving environment (PSE). I offer a simplified application here in the hopes that the general ideas can be useful to someone.
What has also initiated a desire to see this released is my wife’s iPhone with its addictive applications. On the downside, there is a serious lack of real estate and unless I convert to Mac I can’t get my hands on the SDK. However, coupling the iPhone’s portability with a personal server running traditional applications makes for a cheap and powerful PSE. Computational tasks and data storage can take place server side with the iPhone serving as a very pleasing web portal.
There are plenty of applications of these ideas to the sciences, for someone with a modicum of talent and vision. For the purposes of this example I’m going to use free financial data and the computational task will be options pricing under the Black Scholes model for European contracts. I’ve taken a data source at random, and I have no idea if there is a European style options contract available. Also realize that there is potentially unlimited risk associated with options, and it is widely held that the traditional theory is naïve and at worst downright wrong, particularly when using historical data to calculate volatility. So use this example under advisement…and as always, please be careful with curl, don’t get yourself banned 
- Set up MySQL + Apache + PHP on a server
- Create Database + table on server:
//invoke mysql (as root)
>mysql –u root –p
//create database
>create database tech_stock;
//extend permission to user bill with password *****
>GRANT ALL ON tech_stock.* TO bill@localhost IDENTIFIED BY “*****“;
//quit;
>quit;
//as user, invoke mysql as before; select database
>mysql tech_stock –u bill –p
//create a table
>create table tech_A( n_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, date VARCHAR(8), open FLOAT,
high FLOAT, low FLOAT, close FLOAT, volume FLOAT);
//check it exists
>DESCRIBE tech_A;
//logout
>quit;
- Write a shell script to grab data:
#!/bin/sh
# data update/download wjb 02/08
Month=$(date +%b)
Day=$(date +%d)
Year=$(date +%G)
str1=”http://finance.google.com/finance/historical?cid=659815&startdate=Feb+10%2C+2008&enddate=”
str2=”%2C+”$Year”&output=csv”
curl $str1$Month”+”$Day$str2 > tech_A.csv
exit #cheerio
- Make a html portal:
<html><body style=”font-family:verdana”>
<title>PSE Demo</title>
<form action=”main.php” method=”post” enctype=”multipart/form-data”>
<font size=”5″ style=”color:blue”>Data Processing Portal<br>
<table style=”background-color:blue”>
<tr><td><font color=”white”>Password:</font> </td>
<td><input type=”password” name=”pswd” /> </td></tr>
<tr><td><font color=”white”>Email: </font></td>
<td><input type=”text” name=”email” /> </td></tr>
<tr><td><input type=”submit” name=”cmdupload” value=”Submit” /></tr></td>
</table>
</form>
<br>
<font size=”1″>
Collaboratory for SDE data modeling <br>
</font>
</body></html>
more to come…
Comments Off | 
cyber, programming, quantitative finance | Tagged: PHP, Black Scholes, iPhone, distributed problem solving environment, Apache, octave, MySQL | 
Permalink
Posted by bbrouwer
February 3, 2009
I’ve been working with colleagues and collaborators from the UK to mine NMR spectra and corresponding molecular structures from documents. The object is to create an XML/CML database to give researchers unprecedented access to information, useful in (for instance) drug discovery. At this stage, I have focused on writing algorithms for the extraction of molecules to *svg, and NMR data to *txt. The latter is then refined and processed, first to determine peak positions. The data is then optimally fit using mixture models, and peak lists created automatically using standard and novel algorithms.
3 Comments | 
Machine Learning, cyber, document analysis, nuclear magnetic resonance | Tagged: *svg, CML, data mining, mixture models, molecular data, NMR, spectral data, XML | 
Permalink
Posted by bbrouwer
January 14, 2009
Excellent description by Eberhard Engel here
I’m in the process of trying to create a large number of NCPP’s for the calculation of magnetic properties, using GIPAW. The end goal is to try, in conjunction with machine learning and NMR, to do structure determination for complicated materials.
This is somewhat related to the work presented at M’soft eScience, with the slightly different goal of going from a large database of calculated values, and via clustering and comparison to NMR simulations, back-out chemical structures.
Leave a Comment » | Ab Initio, Machine Learning, cyber, nuclear magnetic resonance | Tagged: MQMAS, Norm Conserving, Pseudopotentials, support vector machines, Troullier-Martins | Permalink
Posted by bbrouwer
October 24, 2008
A presentation made to CEKA meeting at Penn State 10/24/08 on the collaboratory project I participate in… covers Knowledge Discovery, Machine Learning, Databases, Web 2.0 etc etc
chemxseer talk
Leave a Comment » | 
Machine Learning, cyber | Tagged: ChemXSeer | 
Permalink
Posted by bbrouwer
September 30, 2008
grids are becoming increasingly popular but after you remove some fancy protocols and security measures they are still composed of distributed machines. If you have a fairly ‘dirty’ grid, ie., very heterogeneous and not endowed with MPI etc, when performing data decomposition, some thought must be given as to how data is divided. Suppose you would like to divide M same tasks amongst N processors. If (taking into account network latency etc) one can order the time it takes to perform a single task as a1 < a2 < .. < ai < .. < aN, then a naive division of labor for each node would be:
But this overlooks possible data collision and latency involved in loading buffers at the manager, so take this into account with an extra term, giving the tasks for a single node as:
where b is less than one, and since the total tasks handled by N procs must be M, this implies the relationship btwn r and b:
r may be determined from experiment, and b follows.
Leave a Comment » | cyber, grid | Tagged: algorithm, distributed, grid, schedule | Permalink
Posted by bbrouwer
September 26, 2008
In an unusual twist, a few news sources are reporting on the aforementioned image proc work at Penn State: New Scientist , Naked Scientists and L’Atelier
I’m flattered and more than a little surprised
ED: also showed up in ACM tech news
2 Comments | Machine Learning, cyber, document analysis | Tagged: image processing, Machine Learning, press | Permalink
Posted by bbrouwer
September 16, 2008
Here’s the paper abstract, links to paper/talk follow:
“Most search engines index the textual content of documents in digital libraries. However, scholarly articles frequently report important findings in figures for visual impact and the contents of these figures are not indexed. These contents are often invaluable to the researcher in various fields, for the purposes of direct comparison with their own work. Therefore, searching for figures and extracting figure data are important problems. To the best of our knowledge, there exists no tool to automatically extract data from figures in digital documents. If we can extract data from these images automatically and store them in a database, an end-user can query and combine data from multiple digital documents simultaneously and efficiently. We propose a framework based on image analysis and machine learning to extract information from 2-D plot images and store them in a database. The proposed algorithm identifies a 2-D plot and extracts the axis labels, legend and the data points from the 2-D plot. We also segregate overlapping shapes that correspond to different data points. We demonstrate performance of individual algorithms, using a combination of generated and real-life images.”
paper
talk
Leave a Comment » | Machine Learning, cyber, document analysis | Tagged: algorithm, image analysis, Machine Learning, pattern recognition | Permalink
Posted by bbrouwer
