tag:blogger.com,1999:blog-42619544314211799082024-03-05T20:53:29.171+00:00One of my most productive days was throwing away 1000 lines of code.
— Ken ThompsonSteve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-4261954431421179908.post-46151918947278863182013-10-19T22:10:00.000+01:002013-10-19T22:10:42.494+01:00Vi Users Have More ControlI was testing the macro focus on my new camera this morning, and thought this might be amusing to the programmers out there.
So, at the risk of starting a text editor flame war...
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIRsgfVI3fhSlpf1CB8VPEndmq1RlZidE2r6M5-_n07p7U5qOdyhPpqJFffNmdFCRNm5n6oBxLGeWHj1I7qhfyppdQp2D1SjbVDTFXX5xUNs_SCM2LKP7cvlN__4TSGU-RGk7tGRo-1Nq7/s1600/control-vi.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIRsgfVI3fhSlpf1CB8VPEndmq1RlZidE2r6M5-_n07p7U5qOdyhPpqJFffNmdFCRNm5n6oBxLGeWHj1I7qhfyppdQp2D1SjbVDTFXX5xUNs_SCM2LKP7cvlN__4TSGU-RGk7tGRo-1Nq7/s400/control-vi.png" /></a></div>Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com0tag:blogger.com,1999:blog-4261954431421179908.post-71890330857334850072013-02-10T14:38:00.000+00:002013-02-10T17:28:49.641+00:00Alert! Testing Javascript's alert function with Jasmine<p>I was refactoring some javascript code at work the other day. Literally, 3000 lines of logic to decide on whether to throw an alert and what to include in the message, if so. I kid you not. There's a serious problem with some of the legacy code where I work, though, admittedly, the logic was a big tricky. All but about 300 lines of that code was pure duplication (copied, more or less line for line across 10 pages). So, by the end of the day I'd reduced the 3000 lines to about 60, and added about 240 lines of <a href="http://pivotal.github.com/jasmine/">Jasmine</a> tests (still 300 lines, but now heavily tested).</p>
<p>Anyway, in doing this, I came up with a way to test the function which threw the potential alert messages. I thought I'd share this idea, as I rather enjoyed coding it. Either way, I wanted to document it for my own reference, and hopefully others will find it useful, too.</p>
<p>The gist of the problem here was that there was a single function call (triggered from clicking a 'Save' button on various pages), which susequently called a number of other functions, depending on the case being tested, each of which had the potential to add something to the alert message. I had refactored these subsequent functions to just return strings with their various contributions, and so those functions were very easy to test individually.</p>
<p>What was tricky, was figuring out how to test the original function that actually throws the alert, depending on the responses. How do you test an alert? I guess there are various ways, but here's what I came up with...</p>
<p>Imagine if you will, the 'main' function is called 'alert_me' which ultimately just returns so that the form post can be submitted. 'alert_me' calls a number of other functions, all of which use a function called 'get_val' to reference the data that they use to calculate their response (I tried to simplify this problem for this post as best I could, so we could just focus on the problem at hand. Hopefully, this makes sense.)</p>
<p>I set up an object to contain default values for the various possible responses of 'get_val' (for this exercise foo, bar, or baz), and a helper function to set them ('set_val'). (Note: I'm not testing 'get_val', but I need to be able to mock its output in order to get the correct response from 'alert_me'). I also use a variable to capture the alert message which I can set by overriding javascript's 'alert' function.</p>
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> describe("alert_me", function() {
var get_val_rtn_val, alert_msg;
var set_val = function(field, val){ get_val_rtn_val[field] = val; };
beforeEach(function() {
get_val_rtn_val = { // reset values
'foo': 0,
'bar': 0,
'baz': 0
};
spyOn(window, 'get_val').andCallFake(function(field) {
return get_val_rtn_val[field];
});
alert_msg = '_default_';
spyOn(window, 'alert').andCallFake(function(msg) {
alert_msg = msg;
});
});
it("tests the test", function(){
expect(get_val('foo')).toEqual(0);
set_val('foo', 1);
expect(get_val('foo')).toEqual(1);
expect(alert_msg).toEqual('_default_');
alert('foobar');
expect(alert_msg).toEqual('foobar');
});
it("contains '_default_' for default case", function(){
alert_me();
expect(alert_msg).toEqual('_default_');
});
it("contains 'Wheezy' for foo=1", function(){
set_val('foo', 1);
alert_me();
expect(alert_msg).toEqual('Please note the following: Wheezy');
});
it("contains 'Dopey' for bar=1, baz=1", function(){
set_val('bar', 1);
set_val('baz', 1);
alert_me();
expect(alert_msg).toEqual('Please note the following: Dopey');
});
it("contains 'Wheezy' and 'Dopey' for foo=1, baz=1", function(){
set_val('foo', 1);
set_val('baz', 1);
alert_me();
expect(alert_msg).toEqual('Please note the following: Wheezy, Dopey');
});
});
</code></pre>
<p>Naturally, if 'alert_me' is working as expected, these tests should pass!</p>
<p>I'd love to hear feedback about this solution. It seemed pretty simple and straightforward to me. I find Jasmine to be flexible and fun to work with.</p>
<p>As a side note, one of the things that I really enjoyed about writing all these tests was that it made me think differently about the way I write Javascript. I believe the refactored code was a lot more elegant, largely due to the fact that it was designed to be easily tested.</p>Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com2tag:blogger.com,1999:blog-4261954431421179908.post-44256971788847934372012-10-21T22:12:00.003+01:002012-10-21T22:36:58.223+01:00A First Look at node.js, A Logging ApplicationI had a chance this week to take a first look at node.js. I'd been meaning to do this at some point, as it's a critical component in my ongoing investigation of <a href="http://blog.q-media.com/2012/08/the-state-of-full-stack-javascript.html">full-stack JavaScript</a>, so I was glad to get this opportunity.
<br />
<br />
I've already covered a bit about <a href="http://blog.q-media.com/2012/09/mongo-22-aggregation-framework-stock.html">MongoDB</a>, but, obviously there's a lot more to go over here. One thing that is worth noting is that <a href="http://www.mongodb.org/">Mongo</a> seems to be emerging as the persistent database of choice in the full-stack frameworks (with <a href="http://redis.io/">Redis</a> often being used for caching). [Just FYI, I'll be at the <a href="http://www.10gen.com/events/mongodb-boston">Mongo Meetup in Boston</a> this coming Wednesday]
<br />
<br />
But, while one has many choices on the datastore side of things, there's one technology that seems to be indispensable for doing full-stack JavaScript, and server-side JavaScript in particular, and that is <a href="http://nodejs.org/">node.js</a>. In short, Node is a JavaScript engine, based on <a href="http://code.google.com/p/v8/">Google's super-fast and open source V8</a>, with particular emphasis on network application capabilities, CommonJS library implementation, and package management (via <a href="https://npmjs.org/">npm</a>). Although it's possible to <a href="http://nodejs.org/api/child_process.html#child_process.fork">fork threads</a>, and there's even a <a href="http://nodejs.org/api/cluster.html">cluster API</a>, a node application is generally designed to work in a single event loop, using non-blocking, asynchronous, I/O to achieve great efficiency in processing requests.
<br />
<br />
Point is, it's <a href="http://zgadzaj.com/benchmarking-nodejs-basic-performance-tests-against-apache-php">fast</a>.
<br />
<br />
Nearly as fast as Java, and orders of magnitude <a href="http://shootout.alioth.debian.org/u64/which-programs-are-fastest.php">faster than Ruby</a>.
<br />
<br />
It's "event-driven," using callback functions to deliver asynchronous functionality, very much like the way JavaScript works in the browser. However, it's a very different way of thinking when programming server-side code. My first pass at understanding it took a little effort. I focused on using the setTimeout function in JavaScript to test some assumptions about how events are dealt with. Initially, I was quite confused because I would trigger two requests in short succession in two Firefox tabs, each of which would fire a 10 second wait using setTimeout, and the requests were served serially, one after the other, taking a total of 20 seconds to process both. However, as it turns out, this was happening because of the way Firefox was managing the requests. Using curl on the command line and executing multiple requests more or less simultaneously revealed that these "waits" are, in fact, running in parallel.
<br />
<br />
This first attempt can be seen here:
<br />
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> var http = require("http");
function do_thing(response, callback){
var d=new Date();
console.log('calling setTimeout '+d.toLocaleTimeString());
var timer = setTimeout(function(){
var d=new Date();
console.log('timeout callback '+d.toLocaleTimeString());
respond(response);
}, 10000);
var d2=new Date();
console.log('setTimeout called '+d2.toLocaleTimeString());
}
function respond(response){
var d=new Date();
console.log('sending response '+d.toLocaleTimeString());
response.writeHead(200, {"Content-Type": "text/plain"});
response.write("hello at "+d.toLocaleTimeString());
response.end();
}
http.createServer(function(request, response) {
var d=new Date();
console.log('got request '+d.toLocaleTimeString());
do_thing(response, respond);
}).listen(8888);
var d=new Date();
console.log('running... '+d.toLocaleTimeString());
// UPDATE: Firing two requests curl in two Terminal windows resolves the issue below
// This code is good. The test (using Firefox) was bad.
//Two web requests in close succession, still have to wait 20 secs for 2nd request to complete...
//OUTPUT AS FOLLOWS:
//
//running... 12:12:00
//got request 12:12:06
//calling setTimeout 12:12:06
//setTimeout called 12:12:06
//timeout callback 12:12:16
//sending response 12:12:16
//got request 12:12:16
//calling setTimeout 12:12:16
//setTimeout called 12:12:16
//timeout callback 12:12:26
//sending response 12:12:26
</code></pre>
<br />
So, the next order of business was to use this knowledge to implement an actual application. The task at hand, a logging server. The basic requirement is this: HTTP POST with some arbitrary data, writes to Mongo database. Any application (client) can post data, in a format of it's own choosing (given a few required fields so that the source could be identified and filtering done). This was essentially fulfilling an "audit" requirement (details unimportant). This is about as simple as it gets, folks, for a real-life application.
<br />
<br />
I initially looked at <a href="http://mongoosejs.com/">Mongoose</a> for interacting with the database. It's a pretty clean interface, and worked well. However, soon after getting that more or less working, I came across this module (which is part of the <a href="https://github.com/flatiron/winston">flatiron framework</a>), a "multi-transport" logging library. This made sense, to use a logging library to do logging. The ability to use various storage mechanisms is very nice, and the fact that it supported Mongo out of the box was great.
<br />
<br />
One thing to note about Mongoose, in case this helps anyone who's looking at it, is that it uses the concept of a Schema. Which a first blush seems kind of silly when dealing with Mongo (which is schemaless, duh). And, in particular, I wanted to have a schemaless "logentry" which could be formatted by the application submitting the data. Fortunately, Mongoose supports a datatype called "Mixed" which essentially let's you pass a Hash, like so:
<br />
<br />
mongoose.model('AuditLog', new Schema({ application_name: String, log_entry: Schema.Types.Mixed, timestamp: Date }));
<br />
<br />
In any case, Mongoose was not part of the final solution. Winston was.
<br />
<br />
There's one other important piece which came up, though, too. And that was the issue of "frameworks." Flatiron, for example, where Winston comes from, is a framework. A bit of Googling, though, and I decided that the framework to use was <a href="http://expressjs.com/">express</a>. It seems to be the node framework with the most traction. It also is very lightweight, and is the framework upon which most of the more fully-formed frameworks seem to be built (i.e. <a href="http://towerjs.org/">Tower.js</a> and <a href="http://derbyjs.com/">Derby</a>, both of which I may try to look at for future applications).
<br />
<br />
Express certainly helped dealing with posted data and routing, which otherwise could have been tricky.
<br />
<br />
Putting all the pieces together, one gets this (complete with URL-based versioning which I chose to implement despite the controversy on this subject):
<br />
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> // requirements
var winston = require('winston');
var MongoDB = require('winston-mongodb').MongoDB;
var express = require('express');
// set up winston
winston.add(MongoDB, {db:'winston'});
winston.remove(winston.transports.Console);
winston.emitErrs = true;
winston.handleExceptions(new winston.transports.File({ filename: 'exceptions.log' }))
// set up express app
var app = express();
// found out the hard way that this middleware is required to parse data
app.use(express.bodyParser());
// routing
// note: urls not implemented will serve 404 not found
app.all('/api/log', function(req, res){
res.status(301).redirect('/api/v1/log');
});
app.get('/api/v1/log', function(req, res){
res.send(500, { error: 'GET not implemented in v1' });
});
app.post('/api/v1/log', function(req, res){
var required = ['appname', 'logname', 'source', 'logentry'];
for(param in required){
if(! req.param(required[param])){
res.send(500, { error: required[param]+' is required' });
return false;
}
}
winston.info('Winston Logging', req.body);
res.send('Thanks for that, '+req.body.appname);
});
// start server
app.listen(8888);
console.log('Listening on port 8888');
</code></pre>
<br />
This must seem incredibly naive, and, indeed, it is. However, it gets the job done. Future plans include responding to GET request to return log data (for querying).
<br />
<br />
Now, one final note: testing. Oh, jeez. This was not a simple thing.
<br />
<br />
It's not clear to me that this aspect of node development is entirely full-formed. I didn't have a lot of luck finding consensus in this area. I chose, in the end, to use <a href="http://vowsjs.org/">Vows</a>. I'm not entirely certain I got this right, and I plan to spend some time fleshing this out at a later date. Problems with this solution, as it currently stands, are: environment management (i.e. write test data to production db?), and the fact that I have to have the Mongo and Node servers running in order to run the tests. Not to mention the fact that I'm not testing the asynchronous code, i.e. callbacks, although this seems like it is possible, I just haven't figured it out yet. I believe there are probably better ways of going about testing a node app, but I am a novice, and have much to learn.
<br />
<br />
For the record, though, here's what I've got for tests:
<br />
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> var vows = require('vows'),
assert = require('assert'),
request = require('request'),
apiUrl = 'http://127.0.0.1:8888';
var apiTest = {
general: function( method, url, data, callback ){
//console.log(data);
request(
{
method: method,
url: apiUrl+(url||''),
followRedirect: false,
json: data || {}
},
function(req, res){
callback( res );
}
)},
get: function( url, data, cb ){ apiTest.general( 'GET', url, data, cb ) },
post: function( url, data, cb ){ apiTest.general( 'POST', url, data, cb ) }
}
function assertStatus(code) {
return function (res, b, c) {
assert.equal(res.statusCode, code);
};
}
var log_entry = { foo: 'bar', baz: 'blah' };
var log_good = { logentry: log_entry, appname: 'my appname', logname: 'my logname', source: 'my source' };
var log_no_appname = { logentry: log_entry, logname: 'my logname', source: 'my source' };
var log_no_logname = { logentry: log_entry, appname: 'my appname', source: 'my source' };
var log_no_source = { logentry: log_entry, appname: 'my appname', logname: 'my logname' };
var log_no_logentry = { appname: 'my appname', logname: 'my logname', source: 'my source' };
// Create a Test Suite
vows.describe('Server').addBatch({
'when doing get request on root url': {
topic: function () { apiTest.get('', log_good, this.callback ) },
'returns error': assertStatus(404)
},
'when doing get request with badly formatted url': {
topic: function () { apiTest.get('/foo', log_good, this.callback ) },
'returns error': assertStatus(404)
},
'when doing get request with default url': {
topic: function () { apiTest.get('/api/log', log_good, this.callback ) },
'returns redirect': assertStatus(302) // note: this actually is a 301, but the test seems to need 302
},
'when doing post with incorrect data: no appname': {
topic: function () { apiTest.post('/api/v1/log', log_no_appname, this.callback ) },
'returns error': assertStatus(500)
},
'when doing post with incorrect data: no logname': {
topic: function () { apiTest.post('/api/v1/log', log_no_logname, this.callback ) },
'returns error': assertStatus(500)
},
'when doing post with incorrect data: no source': {
topic: function () { apiTest.post('/api/v1/log', log_no_source, this.callback ) },
'returns error': assertStatus(500)
},
'when doing post with incorrect data: no logentry': {
topic: function () { apiTest.post('/api/v1/log', log_no_logentry, this.callback ) },
'returns error': assertStatus(500)
},
'when doing post with correct data': {
topic: function () { apiTest.post('/api/v1/log', log_good, this.callback ) },
'is success': assertStatus(200)
}
}).run(); // Run it
</code></pre>
<br />
And that's it! (Other than the package.json file which handles dependency management via npm, similar to the way that a Gemfile and bundle does in Rails.)
<br />
<br />
Not much more to say, really, other than that this was a fun and interesting first crack at node.js, which I hope to get an opportunity to work more with in future.
<br />
<br />
I look forward to continuing to get to grips with full-stack JavaScript, and I hope to be able to implement a "real" application soon, and will blog about it here. In the meantime, and, in lieu of gifts, please enjoy this <a href="http://notinventedhe.re/on/2011-7-26">comic</a>.
<br />
<br />
Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com0tag:blogger.com,1999:blog-4261954431421179908.post-72570906240292020132012-09-22T20:03:00.000+01:002012-11-06T11:10:22.837+00:00Mongo 2.2 Aggregation Framework: Stock Price Data<br />
Just wanted to give a shout out to the <b>MongoDB</b> (<a href="http://www.mongodb.org/">http://www.mongodb.org</a>) folks for the new <b>Aggregation Framework</b> in version 2.2 (which thankfully was officially marked as stable about three weeks ago).<br />
<br />
I've been using Mongo to build an ad-hoc reporting (real-time basic statistics with arbitrary filters, etc) app at work, and had previously been using Map Reduce to do realtime data aggregation, much to my pain and anguish. To be fair, Map Reduce was not designed to work in real-time, but I found that it worked well enough on small datasets. When the data numbered in the tens of thousands, however, speed became a real problem.<br />
<br />
The nice thing about Map Reduce is that it all takes place in the database. You can just read your results from your target collection when the computation is done. Mechanically, this is a great way to go. But if you simply need your computations to happen in real-time, then you're stuck with bad performance.<br />
<br />
Benchmarking (at least for this application) revealed about a five times speed improvement using the Aggregation Framework. This meant that operations on datasets of 10-20K records, doing calculations on 100-200 values (so, getting into the millions of data points) could actually happen in real-time. A late night of code refactoring, and problem solved. [Note: in case you're wondering how to get 'round the 16MB doc size limit for this, I set up "batch processing" of values, so, for example in Ruby: variable_list.each_slice(10){ ... }].<br />
<br />
One thing that's quite different about using this new framework, is that the computation happens in the application code (in this case Ruby). The aggregation operations are only being used to extract the data for processing. With Map Reduce, you can write arbitrarily complex functions to operate on your data. With the Aggregation Framework, this has to happen in your application (to be overly simplistic, think of aggregation as the map stage, while the reduce/finalize is done in the client -- at least that's true in our particular circumstance, as we have to operate on the data in ways that the Aggregation Framework itself does not allow). So, no doubt, in a year or two (as our database grows), we will have to look into other mechanisms for optimization. But this is great for now.<br />
<br />
I wanted to do a quick demo of the new features, so I turned to my stock price data collection. Basically, I have a script that's been collecting intra-day trading info on the Dow Jones stocks from Yahoo Finance. It's been running about over a year and a half, so there's a lot of data there, and it's quite useful for exercises such as this.<br />
<br />
The scripts I used for this demo can be found on github (unfortunately, the data itself is quite large, so I did not include it this time): <b><a href="https://github.com/sqpierce/stocks">https://github.com/sqpierce/stocks</a></b><br />
<br />
Here's a rundown of the components:<br />
<br />
<br />
<ul>
<li>Firstly, there's a script to grab the data (this is being run by cron each minute):</li>
<li>Second, there's a Ruby script I just wrote to convert all the data files into a single JSON file for importing into Mongo</li>
<li>Last, here are some basic Aggregation Framework operations, just written in JavaScript and executed via the Mongo shell (for now)</li>
</ul>
<br />
<b>SHELL SCRIPT TO GET DATA</b><br />
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> #!/bin/bash
# has been running approx 1.5 yrs every minute while NYSE is open
DIR=~/projects/stocks/data
hour=`date -u +%H`
day=`date -u +%u`
file=$DIR/`date -u +%F`
# Dow Jones stock symbols
syms="MMM+AA+AXP+T+BAC+BA+CAT+CVX+KO+CSCO+DIS+DD+XOM+GE+HPQ+HD+IBM+INTC+JNJ+JPM+KFT+MCD+MRK+MSFT+PFE+PG+TRV+UTX+VZ+WMT"
# Yahoo finance stock format: http://dirk.eddelbuettel.com/code/yahooquote.html
#symbol, name, last trade date, last trade time, last trade price, dividend yield, P/E ratio, volume, day's value change, previous close
fmt="snd1t1l1yrvw1p"
url="http://download.finance.yahoo.com/d/quotes.csv?s="$syms"&f="$fmt
log (){
echo "#("`date -u`")" $1 >>$file
}
dostock (){
log "file "$file" hour "$hour" day "$day" dir "$DIR
log "using command: curl $url"
curl $url >>$file
}
# note that $hour throws error when 08 or 09 because interpreted as octal - need to investigate
# just changing to 8 or 9 as string using below doesn't seem to fix problem (or, rather, creates another problem
# hour=`date -u +%H`|sed 's/^0*//'
# doesn't matter as we're using UTC, so those hours are out of range
if (( $hour > 12 && $hour < 23 && $day < 6 ))
then
dostock
fi
</code></pre><b>RUBY SCRIPT TO MUNGE DATA</b><br />
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> # utility to munge data into single file for insertion into Mongo
# ruby to_json.rb > stock_data.json 2> to_json.log
require 'csv'
DEBUG=false
dir='data'
count = 0
Dir.entries(dir).each do |file|
next if file =~ /\./ # skip directories
File.readlines("#{dir}/#{file}").each do |line|
next if line =~ /^\s*#/ # skip comments
break if count > 10 && DEBUG # testing only (read 10 lines)
@fields = CSV.parse(line)[0]
warn @fields.inspect if DEBUG # to STDOUT
time_str = @fields.slice!(2,2).join(' ')
warn time_str if DEBUG
begin # NOTE: some of the data has badly formatted dates
date = DateTime.strptime(time_str, '%m/%d/%Y %H:%M%P')
warn date.strftime '%Y-%m-%d %H:%M' if DEBUG
secs = date.to_time.to_i * 1000 # NOTE: must convert seconds to milliseconds
rescue
warn "#{file} - #{time_str} - #{line}" # prints to STDERR
next
end
begin
match = @fields[6].match /([+-]*\d+\.\d+)/
change = match[1].to_f # convert change to float
warn "change: #{@fields[6]} -> #{match.inspect} -> #{change}" if DEBUG
rescue
warn "no match for change on: #{line}"
change = nil
end
# NOTE: $date must be in double quotes
puts "{ symbol: \"#{@fields[0]}\", name: \"#{@fields[1]}\", price: #{@fields[2].to_f}, dividend_yield: #{@fields[3].to_f}, p_e_ratio: #{@fields[4].to_f}, volume: #{@fields[5].to_i}, day_value_change: #{change}, previous_close: #{@fields[7].to_f}, last_trade_time: { \"$date\": #{secs} } }" # json to stdout
count+=1
end
break if DEBUG # testing only (quit after first file)
end
</code></pre>
<b>MONGO AGGREGATION JAVASCRIPT</b><br />
<pre style="font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;background-image:URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif);padding:0px;color:#000000;text-align:left;line-height:20px;"><code style="color:#000000;word-wrap:normal;"> // # mongod --nojournal
// # mongoimport --db stocks --collection price --type json --file stock_data.json
// import looks good
> db.price.find()
{ "_id" : ObjectId("505dded4062e11e40afc5c46"), "symbol" : "AXP", "name" : "American Express ", "price" : 44.73, "dividend_yield" : 2.01, "p_e_ratio" : 14.6, "volume" : 500, "day_value_change" : 0, "previous_close" : 44.73, "last_trade_time" : ISODate("2011-01-06T16:00:00Z") }
{ "_id" : ObjectId("505dded4062e11e40afc5c48"), "symbol" : "BAC", "name" : "Bank of America C", "price" : 14.44, "dividend_yield" : 0.28, "p_e_ratio" : 0, "volume" : 2942961, "day_value_change" : 0, "previous_close" : 14.44, "last_trade_time" : ISODate("2011-01-06T16:02:00Z") }
{ "_id" : ObjectId("505dded4062e11e40afc5c49"), "symbol" : "BA", "name" : "Boeing Company (T", "price" : 68.8, "dividend_yield" : 2.44, "p_e_ratio" : 14.9, "volume" : 2810, "day_value_change" : 0, "previous_close" : 68.8, "last_trade_time" : ISODate("2011-01-06T16:01:00Z") }
{ "_id" : ObjectId("505dded4062e11e40afc5c4a"), "symbol" : "CAT", "name" : "Caterpillar, Inc.", "price" : 93.54, "dividend_yield" : 1.84, "p_e_ratio" : 30.74, "volume" : 2100, "day_value_change" : 0, "previous_close" : 93.54, "last_trade_time" : ISODate("2011-01-06T16:01:00Z") }
{ "_id" : ObjectId("505dded4062e11e40afc5c4b"), "symbol" : "CVX", "name" : "Chevron Corporati", "price" : 90.69, "dividend_yield" : 3.13, "p_e_ratio" : 10.84, "volume" : 100, "day_value_change" : 0, "previous_close" : 90.69, "last_trade_time" : ISODate("2011-01-06T16:00:00Z") }
// list symbols
> db.price.aggregate({"$group": {_id: 0, symbols: {"$addToSet": "$symbol"}}});
{
"result" : [
{
"_id" : 0,
"symbols" : [
"WMT",
"UTX",
"DD",
"AXP",
"TRV",
"PFE",
"MRK",
"MCD",
"KFT",
"VZ",
"XOM",
"JPM",
"INTC",
"IBM",
"AA",
"HD",
"MSFT",
"T",
"DIS",
"BAC",
"CSCO",
"HPQ",
"MMM",
"CVX",
"CAT",
"KO",
"PG",
"GE",
"JNJ",
"BA"
]
}
],
"ok" : 1
}
> db.price.ensureIndex({symbol:1})
> db.price.aggregate({"$project": {symbol: 1}}, {"$group": {_id: 0, symbols: {"$addToSet": "$symbol"}}});
//neither the index nor the project tag sped up this query (more or less the same)
> db.price.aggregate({"$project": {symbol: 1}}, {"$group": {_id: "$symbol", counts: {"$sum": 1}}});
// list counts per symbol in whole collection
> db.price_from_csv.aggregate({"$project": {symbol: 1}}, {"$group": {_id: "$symbol", counts: {"$sum": 1}}});
{
"result" : [
{
"_id" : "WMT",
"counts" : 263888
},
{
"_id" : "UTX",
"counts" : 263888
},
{
"_id" : "DD",
"counts" : 263888
},
{
"_id" : "AXP",
"counts" : 263888
},
{
"_id" : "TRV",
"counts" : 263888
},
{
"_id" : "PFE",
"counts" : 263888
},
{
"_id" : "MRK",
"counts" : 263888
},
{
"_id" : "MCD",
"counts" : 263888
},
{
"_id" : "KFT",
"counts" : 263888
},
{
"_id" : "VZ",
"counts" : 263888
},
{
"_id" : "XOM",
"counts" : 263888
},
{
"_id" : "JPM",
"counts" : 263888
},
{
"_id" : "INTC",
"counts" : 263888
},
{
"_id" : "IBM",
"counts" : 263888
},
{
"_id" : "AA",
"counts" : 263888
},
{
"_id" : "HD",
"counts" : 263888
},
{
"_id" : "MSFT",
"counts" : 263888
},
{
"_id" : "T",
"counts" : 263888
},
{
"_id" : "DIS",
"counts" : 263888
},
{
"_id" : "BAC",
"counts" : 263888
},
{
"_id" : "CSCO",
"counts" : 263888
},
{
"_id" : "HPQ",
"counts" : 263888
},
{
"_id" : "MMM",
"counts" : 263888
},
{
"_id" : "CVX",
"counts" : 263888
},
{
"_id" : "CAT",
"counts" : 263888
},
{
"_id" : "KO",
"counts" : 263888
},
{
"_id" : "PG",
"counts" : 263888
},
{
"_id" : "GE",
"counts" : 263888
},
{
"_id" : "JNJ",
"counts" : 263888
},
{
"_id" : "BA",
"counts" : 262690
}
],
"ok" : 1
}
// list counts per year for a symbol (obviously some bad dates got into the mix, interesting)
> db.price.aggregate({"$match": {symbol: "BA"}}, {"$project": {last_trade_time: 1}}, {"$group": {_id: {"$year": "$last_trade_time"}, count: {"$sum": 1}}});
{
"result" : [
{
"_id" : 2012,
"count" : 111276
},
{
"_id" : 1970,
"count" : 2
},
{
"_id" : 2011,
"count" : 151412
}
],
"ok" : 1
}
// time for an index
> db.price.ensureIndex({last_trade_time:1})
// same again for all
> db.price.aggregate({"$project": {symbol: 1, last_trade_time: 1}}, {"$group": {_id: {"$year": "$last_trade_time"}, count: {"$sum": 1}}});
{
"result" : [
{
"_id" : 2012,
"count" : 3338278
},
{
"_id" : 1970,
"count" : 48
},
{
"_id" : 2011,
"count" : 4577116
}
],
"ok" : 1
}
// look for the bad rows (1970)
> db.price.find({ last_trade_time: {"$lt":ISODate("2010-01-01T00:00:00.000Z")} })
{ "_id" : ObjectId("505ddfa6062e11e40a032916"), "symbol" : "MMM", "name" : "ompany Common Sto", "price" : 223658800, "dividend_yield" : 2.56, "p_e_ratio" : 43.21, "volume" : 3498040, "day_value_change" : 90.69, "previous_close" : 90.75, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa6062e11e40a032934"), "symbol" : "MMM", "name" : "ompany Common Sto", "price" : 223658800, "dividend_yield" : 2.56, "p_e_ratio" : 43.21, "volume" : 3498040, "day_value_change" : 90.69, "previous_close" : 90.75, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d7d"), "symbol" : "T", "name" : " Inc.", "price" : 223837152, "dividend_yield" : 21, "p_e_ratio" : 16.86, "volume" : 25700700, "day_value_change" : 28.43, "previous_close" : 28.5, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d7f"), "symbol" : "BA", "name" : "ng Company (The) ", "price" : 223842096, "dividend_yield" : 3.18, "p_e_ratio" : 43.25, "volume" : 5961360, "day_value_change" : 72.46, "previous_close" : 72.66, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d84"), "symbol" : "DIS", "name" : " Disney Company (", "price" : 223834224, "dividend_yield" : 2.12, "p_e_ratio" : 108.7, "volume" : 10490500, "day_value_change" : 43.38, "previous_close" : 43.48, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d85"), "symbol" : "DD", "name" : " du Pont de Nemou", "price" : 223828288, "dividend_yield" : 4.14, "p_e_ratio" : 44.41, "volume" : 6414850, "day_value_change" : 54.52, "previous_close" : 54.62, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d86"), "symbol" : "XOM", "name" : "n Mobil Corporati", "price" : 223838976, "dividend_yield" : 2.54, "p_e_ratio" : 47.39, "volume" : 20538500, "day_value_change" : 82.65, "previous_close" : 83.41, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d88"), "symbol" : "HPQ", "name" : "ett-Packard Compa", "price" : 223828416, "dividend_yield" : 1.35, "p_e_ratio" : 152.53, "volume" : 17942300, "day_value_change" : 48.52, "previous_close" : 48.81, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d89"), "symbol" : "HD", "name" : " Depot, Inc. (The", "price" : 223840032, "dividend_yield" : 6.81, "p_e_ratio" : 39.36, "volume" : 11546200, "day_value_change" : 37.1, "previous_close" : 37.155, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d8b"), "symbol" : "INTC", "name" : "l Corporation", "price" : 223838720, "dividend_yield" : 13.8, "p_e_ratio" : 33.24, "volume" : 54533600, "day_value_change" : 21.73, "previous_close" : 21.77, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d93"), "symbol" : "PG", "name" : "ter & Gamble Comp", "price" : 223838448, "dividend_yield" : 4.6, "p_e_ratio" : 33.65, "volume" : 10235700, "day_value_change" : 64.71, "previous_close" : 64.88, "last_trade_time" : ISODate("1970-01-01T19:01:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d80"), "symbol" : "CAT", "name" : "rpillar, Inc. Com", "price" : 223833680, "dividend_yield" : 1.68, "p_e_ratio" : 59.05, "volume" : 5715380, "day_value_change" : 102.38, "previous_close" : 102.75, "last_trade_time" : ISODate("1970-01-01T19:02:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d82"), "symbol" : "KO", "name" : "-Cola Company (Th", "price" : 223835840, "dividend_yield" : 4.35, "p_e_ratio" : 36.17, "volume" : 9558050, "day_value_change" : 63.47, "previous_close" : 63.66, "last_trade_time" : ISODate("1970-01-01T19:02:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d8c"), "symbol" : "JNJ", "name" : "son & Johnson Com", "price" : 223829632, "dividend_yield" : 5.66, "p_e_ratio" : 28.99, "volume" : 10984400, "day_value_change" : 60.9, "previous_close" : 61.16, "last_trade_time" : ISODate("1970-01-01T19:02:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033da0"), "symbol" : "KO", "name" : "-Cola Company (Th", "price" : 223835840, "dividend_yield" : 4.35, "p_e_ratio" : 36.17, "volume" : 9558050, "day_value_change" : 63.47, "previous_close" : 63.66, "last_trade_time" : ISODate("1970-01-01T19:02:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d95"), "symbol" : "UTX", "name" : "ed Technologies C", "price" : 223840112, "dividend_yield" : 2.36, "p_e_ratio" : 50.08, "volume" : 3661150, "day_value_change" : 85.1, "previous_close" : 85.13, "last_trade_time" : ISODate("1970-01-01T19:03:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d81"), "symbol" : "CVX", "name" : "ron Corporation C", "price" : 223829360, "dividend_yield" : 3.02, "p_e_ratio" : 34.21, "volume" : 7859110, "day_value_change" : 96.4, "previous_close" : 97.17, "last_trade_time" : ISODate("1970-01-01T19:04:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d7b"), "symbol" : "AA", "name" : "a Inc. Common Sto", "price" : 223827792, "dividend_yield" : 4.03, "p_e_ratio" : 144.92, "volume" : 28247700, "day_value_change" : 17.22, "previous_close" : 17.39, "last_trade_time" : ISODate("1970-01-01T19:06:00Z") }
{ "_id" : ObjectId("505ddfa7062e11e40a033d90"), "symbol" : "MRK", "name" : "k & Company, Inc.", "price" : 223836448, "dividend_yield" : 13.91, "p_e_ratio" : 21.77, "volume" : 18065100, "day_value_change" : 33.02, "previous_close" : 33.085, "last_trade_time" : ISODate("1970-01-01T19:06:00Z") }
{ "_id" : ObjectId("505de199062e11e40a210d32"), "symbol" : "MMM", "name" : "ompany Common Sto", "price" : 505544928, "dividend_yield" : 2.56, "p_e_ratio" : 42.99, "volume" : 3294730, "day_value_change" : 92.32, "previous_close" : 92.43, "last_trade_time" : ISODate("1970-01-01T20:01:00Z") }
> db.price.remove({ last_trade_time: {"$lt":ISODate("2010-01-01T00:00:00.000Z")} })
// multi-key group and sort
> db.price.aggregate({"$project": {symbol: 1, last_trade_time: 1}}, {"$group": {_id: {symbol: "$symbol", year: {"$year": "$last_trade_time"}}, count: {"$sum":1}}}, {"$sort": {_id: 1}});
{
"result" : [
{
"_id" : {
"symbol" : "AA",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "AA",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "AXP",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "AXP",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "BA",
"year" : 2011
},
"count" : 151412
},
{
"_id" : {
"symbol" : "BA",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "BAC",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "BAC",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "CAT",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "CAT",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "CSCO",
"year" : 2011
},
"count" : 152612
},
{
"_id" : {
"symbol" : "CSCO",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "CVX",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "CVX",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "DD",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "DD",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "DIS",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "DIS",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "GE",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "GE",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "HD",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "HD",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "HPQ",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "HPQ",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "IBM",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "IBM",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "INTC",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "INTC",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "JNJ",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "JNJ",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "JPM",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "JPM",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "KFT",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "KFT",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "KO",
"year" : 2011
},
"count" : 152609
},
{
"_id" : {
"symbol" : "KO",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "MCD",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "MCD",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "MMM",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "MMM",
"year" : 2012
},
"count" : 111275
},
{
"_id" : {
"symbol" : "MRK",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "MRK",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "MSFT",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "MSFT",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "PFE",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "PFE",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "PG",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "PG",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "T",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "T",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "TRV",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "TRV",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "UTX",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "UTX",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "VZ",
"year" : 2011
},
"count" : 152612
},
{
"_id" : {
"symbol" : "VZ",
"year" : 2012
},
"count" : 111275
},
{
"_id" : {
"symbol" : "WMT",
"year" : 2011
},
"count" : 152611
},
{
"_id" : {
"symbol" : "WMT",
"year" : 2012
},
"count" : 111276
},
{
"_id" : {
"symbol" : "XOM",
"year" : 2011
},
"count" : 152610
},
{
"_id" : {
"symbol" : "XOM",
"year" : 2012
},
"count" : 111276
}
],
"ok" : 1
}
// using match (2012 only)
> db.price.aggregate({"$match": {last_trade_time: {"$gte":ISODate("2012-01-01T00:00:00.000Z")}}}, {"$group": {_id: "$symbol", count: {"$sum": 1}}});
{
"result" : [
{
"_id" : "MMM",
"count" : 111275
},
{
"_id" : "UTX",
"count" : 111276
},
{
"_id" : "DD",
"count" : 111276
},
{
"_id" : "AXP",
"count" : 111276
},
{
"_id" : "TRV",
"count" : 111276
},
{
"_id" : "PFE",
"count" : 111276
},
{
"_id" : "MRK",
"count" : 111276
},
{
"_id" : "MCD",
"count" : 111276
},
{
"_id" : "KFT",
"count" : 111276
},
{
"_id" : "VZ",
"count" : 111275
},
{
"_id" : "XOM",
"count" : 111276
},
{
"_id" : "JPM",
"count" : 111276
},
{
"_id" : "INTC",
"count" : 111276
},
{
"_id" : "IBM",
"count" : 111276
},
{
"_id" : "AA",
"count" : 111276
},
{
"_id" : "HD",
"count" : 111276
},
{
"_id" : "MSFT",
"count" : 111276
},
{
"_id" : "BAC",
"count" : 111276
},
{
"_id" : "CSCO",
"count" : 111276
},
{
"_id" : "T",
"count" : 111276
},
{
"_id" : "DIS",
"count" : 111276
},
{
"_id" : "HPQ",
"count" : 111276
},
{
"_id" : "WMT",
"count" : 111276
},
{
"_id" : "PG",
"count" : 111276
},
{
"_id" : "GE",
"count" : 111276
},
{
"_id" : "JNJ",
"count" : 111276
},
{
"_id" : "CVX",
"count" : 111276
},
{
"_id" : "CAT",
"count" : 111276
},
{
"_id" : "KO",
"count" : 111276
},
{
"_id" : "BA",
"count" : 111276
}
],
"ok" : 1
}
// here's one that's actually a bit useful: Bank of America high/low/average price by week in 2012
> db.price.aggregate({"$match": {last_trade_time: {"$gte":ISODate("2012-01-01T00:00:00.000Z")}, symbol: "BA"}}, {"$group": {_id: {"$week":"$last_trade_time"}, high: {"$max": "$price"}, low: {"$min": "$price"}, avg: {"$avg": "$price"}}}, {"$sort":{_id:1}});
{
"result" : [
{
"_id" : 1,
"high" : 74.98,
"low" : 72.79,
"avg" : 73.99888326980202
},
{
"_id" : 2,
"high" : 75.68,
"low" : 74.17,
"avg" : 74.78443128234832
},
{
"_id" : 3,
"high" : 75.981,
"low" : 74.83,
"avg" : 75.4167605833319
},
{
"_id" : 4,
"high" : 76.312,
"low" : 72.96,
"avg" : 75.23588637120919
},
{
"_id" : 5,
"high" : 76.73,
"low" : 73.69,
"avg" : 75.07820737403995
},
{
"_id" : 6,
"high" : 75.9,
"low" : 74.325,
"avg" : 75.17866743023808
},
{
"_id" : 7,
"high" : 75.96,
"low" : 74.78,
"avg" : 75.24222214802415
},
{
"_id" : 8,
"high" : 76.636,
"low" : 75.09,
"avg" : 75.92514222592771
},
{
"_id" : 9,
"high" : 75.76,
"low" : 74.661,
"avg" : 75.09111806064543
},
{
"_id" : 10,
"high" : 74.9501,
"low" : 72.3,
"avg" : 73.6226790136067
},
{
"_id" : 11,
"high" : 75.79,
"low" : 73.2,
"avg" : 74.7180465488498
},
{
"_id" : 12,
"high" : 75.68,
"low" : 73.39,
"avg" : 74.69681463821372
},
{
"_id" : 13,
"high" : 75.27,
"low" : 73,
"avg" : 74.48685070187024
},
{
"_id" : 14,
"high" : 75.4645,
"low" : 73.3,
"avg" : 74.13860771929662
},
{
"_id" : 15,
"high" : 73.77,
"low" : 70.6,
"avg" : 72.4210965729701
},
{
"_id" : 16,
"high" : 74.36,
"low" : 72.325,
"avg" : 73.49481252923461
},
{
"_id" : 17,
"high" : 77.54,
"low" : 72.17,
"avg" : 75.31753740765522
},
{
"_id" : 18,
"high" : 77.82,
"low" : 75.52,
"avg" : 76.7284854496451
},
{
"_id" : 19,
"high" : 76.04,
"low" : 73.28,
"avg" : 74.55083031416956
},
{
"_id" : 20,
"high" : 73.76,
"low" : 68.94,
"avg" : 71.73182498332316
},
{
"_id" : 21,
"high" : 72.18,
"low" : 69.67,
"avg" : 70.88400550458809
},
{
"_id" : 22,
"high" : 70.4825,
"low" : 67.1999,
"avg" : 69.18734449999921
},
{
"_id" : 23,
"high" : 70.3,
"low" : 66.84,
"avg" : 68.64987898599199
},
{
"_id" : 24,
"high" : 72.79,
"low" : 69.84,
"avg" : 71.68434567901214
},
{
"_id" : 25,
"high" : 73.59,
"low" : 71.1,
"avg" : 72.26916556666535
},
{
"_id" : 26,
"high" : 74.36,
"low" : 70.5,
"avg" : 71.82509259752993
},
{
"_id" : 27,
"high" : 74.74,
"low" : 72.5,
"avg" : 73.90557605868787
},
{
"_id" : 28,
"high" : 75.02,
"low" : 70.92,
"avg" : 72.89437934022011
},
{
"_id" : 29,
"high" : 75.07,
"low" : 72.0631,
"avg" : 73.68273189856532
},
{
"_id" : 30,
"high" : 75.93,
"low" : 71.37,
"avg" : 73.76041403333438
},
{
"_id" : 31,
"high" : 75.55,
"low" : 71.23,
"avg" : 73.37943595468138
},
{
"_id" : 32,
"high" : 74.69,
"low" : 72.72,
"avg" : 74.03845807204802
},
{
"_id" : 33,
"high" : 74.43,
"low" : 72.76,
"avg" : 73.7013437333349
},
{
"_id" : 34,
"high" : 74.325,
"low" : 70.04,
"avg" : 72.34569156666593
},
{
"_id" : 35,
"high" : 71.95,
"low" : 70.53,
"avg" : 71.3327890464269
},
{
"_id" : 36,
"high" : 73.25,
"low" : 70.4199,
"avg" : 71.99506040016878
},
{
"_id" : 37,
"high" : 72.39,
"low" : 70.4651,
"avg" : 71.27986476666649
},
{
"_id" : 38,
"high" : 70.7,
"low" : 69.04,
"avg" : 69.96584609175824
}
],
"ok" : 1
}
// use sorting to find extremes
// high by week
> db.price.aggregate({"$match": {last_trade_time: {"$gte":ISODate("2012-01-01T00:00:00.000Z")}, symbol: "BA"}}, {"$group": {_id: {"$week":"$last_trade_time"}, high: {"$max": "$price"}, low: {"$min": "$price"}, avg: {"$avg": "$price"}}}, {"$sort":{high:-1}}, {"$group": {_id: 0, high: {"$first":"$high"}, week: {"$first": "$_id"}}});
{
"result" : [
{
"_id" : 0,
"high" : 77.82,
"week" : 18
}
],
"ok" : 1
}
// low by week
> db.price.aggregate({"$match": {last_trade_time: {"$gte":ISODate("2012-01-01T00:00:00.000Z")}, symbol: "BA"}}, {"$group": {_id: {"$week":"$last_trade_time"}, high: {"$max": "$price"}, low: {"$min": "$price"}, avg: {"$avg": "$price"}}}, {"$sort":{low:1}}, {"$group": {_id: 0, low: {"$first":"$low"}, week: {"$first": "$_id"}}});
{
"result" : [
{
"_id" : 0,
"low" : 66.84,
"week" : 23
}
],
"ok" : 1
}
</code></pre>
<b>CONCLUSION</b><br />
<br />
Each element in the aggregation pipeline is a further transformation of the data. At each stage's output, documents are passed into the next stage's input.<br />
<br />
Obviously, there are probably many more interesting queries one could concoct, but hopefully this gives a reasonable overview of the concept of using the new Aggregation Framework.<br />
<br />
I hope this is helpful to anyone looking at playing around with Mongo in general, or the Aggregation Framework in particular.<br />
<br />
<br />
<br />
<br />Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com0tag:blogger.com,1999:blog-4261954431421179908.post-43808268565606019762012-08-14T19:17:00.002+01:002013-11-30T15:38:35.757+00:00The State of Full-Stack JavaScript[tl;dr: A brief look at JavaScript-based frameworks.]<br />
<br />
<div class="p1">
I was having a conversation with a guy at work the other day, and I suggested (only somewhat jokingly) that the future of web development would be JavaScript-based, not just client-side, but server-side and database, too. I kind of surprised myself by being somewhat excited about the prospect.</div>
<div class="p1">
<br />
I, like most developers I know, learned to love JavaScript (again?) through jQuery <a href="http://jquery.com/">http://jquery.com/</a>. jQuery made writing client-side code fun, similar to the way that Ruby On Rails <a href="http://rubyonrails.org/">http://rubyonrails.org/</a> made writing server-side code fun.</div>
<div class="p1">
<br />
We recently started using MongoDB at work for a specific project <a href="http://www.mongodb.org/">http://www.mongodb.org/</a>. We’re calling the project the “Analytics Engine.” It was originally called “Ad-hoc Reporting.” It’s actually very similar to a project I did years ago while working in the Academic Computing department at Dartmouth College. That project was for an undergrad course called “War and Peace in the 20th Century,” and it was meant to introduce students to basic data analysis concepts by allowing them to choose a couple of variables from a dataset and get some statistical information about the individual variables as well as the relationship between them.</div>
<div class="p1">
<br />
You can have a look here: <a href="http://dataanalysis.q-media.com/">http://dataanalysis.q-media.com/</a></div>
<div class="p1">
<br />
It’s using a database of terrorist incidents which had come from the Rand Corporation (before Rand had set up their own site for doing this type of analysis). <a href="http://www.rand.org/nsrd/projects/terrorism-incidents.html">http://www.rand.org/nsrd/projects/terrorism-incidents.html</a></div>
<div class="p1">
<br />
Please bear in mind that this was just a prototype, and it was also my first ever Rails app (Spring 2006). It’s primative. The full-blown application never got built because the professor who was sponsoring it moved to Stanford shortly thereafter. I’m honestly surprised it still works at all. It’s got a sqlite backend, and it’s based on the idea that all the variables are either categorical or numeric (we left out time-series data, except for filters, as “Year” was entered as a categorical variable).</div>
<div class="p1">
<br />
Anyway, the upshot of all that is that it turns out that databases can be fun, too! MongoDB has proved to be a really slick way to implement this kind of data analysis application. Because the data that we’re doing reporting on comes from a forms-based input system (built on Oracle APEX, don’t ask!), we had to be able to easily handle changes in the schema. Presto! Just don’t use a schema!</div>
<div class="p1">
<br />
So, back to JavaScript (which is, after all, the point of this post)…</div>
<div class="p1">
<br />
Some time ago, not long after that Terrorism application was built, people started to contemplate something outrageous: that JavaScript could (and should) be taken seriously.</div>
<div class="p3">
<span class="s1"><br /></span>
<span class="s1">We had Steve Yegge talking about the “NBL” in February, 2007: <span class="s2"><a href="http://steve-yegge.blogspot.com/2007/02/next-big-language.html">http://steve-yegge.blogspot.com/2007/02/next-big-language.html</a></span></span></div>
<div class="p1">
<br />
Who knew he was talking about JavaScript at the time? C’mon, be honest!</div>
<div class="p1">
<br />
We had JavaScript: The Good Parts by Douglas Crockford (© 2008): <a href="http://www.amazon.com/JavaScript-Good-Parts-Douglas-Crockford/dp/0596517742">http://www.amazon.com/JavaScript-Good-Parts-Douglas-Crockford/dp/0596517742</a></div>
<div class="p1">
(see also this Google Tech Talk: <a href="http://www.youtube.com/watch?v=hQVTIJBZook">http://www.youtube.com/watch?v=hQVTIJBZook</a>)</div>
<div class="p1">
<br />
Now, we even have the (shock, horror) idea that MongoDB might begin to be taken seriously: <span class="s3"><a href="http://www.theregister.co.uk/2012/06/11/mongo_db/">http://www.theregister.co.uk/2012/06/11/mongo_db/</a></span></div>
<div class="p1">
<br />
Other developments worth mentioning: CommonJS promises a JavaScript standard library[<a href="http://www.commonjs.org/">http://www.commonjs.org/</a>], Rhino [<a href="https://developer.mozilla.org/en-US/docs/Rhino">https://developer.mozilla.org/en-US/docs/Rhino</a>], which has been around for a while, provides interesting possibilities for Java integration, and, of course, Node.js [<a href="http://nodejs.org/">http://nodejs.org/</a>], which is built on top of Google’s blazingly fast V8 JS engine, is the cat’s pajamas (according to many: <a href="http://notinventedhe.re/on/2011-7-26">http://notinventedhe.re/on/2011-7-26</a>) for programming network services.<br />
<br />
See the benchmarks on V8 JS here: <a href="http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=all&lang2=v8">http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=all&lang2=v8</a></div>
<div class="p1">
<br />
So, what does it really take to put together a full-stack of piping hot JavaScript pancakes?</div>
<div class="p1">
<br />
Well, first of all, to be fair, I’m slightly misusing the expression “full-stack.” Full-stack can mean Model (ORM), View (templates), and Controller, and Rails qualifies as “full-stack” by some definitions. I’m using a broader interpretation to mean not just the server-side framework, but the database interface language (where SQL served in the past), and, of course, client-side code (where JavaScript is the de facto standard).</div>
<div class="p1">
<br />
So, we’re looking for the full banana. And, when I ask the question, “What is the state of full-stack JavaScript development today?” (to my friend, Google), what I get back, is this (more than two years old): <a href="http://jimbojw.com/fullstack/"><span class="s3">http://jimbojw.com/fullstack/</span></a> which gives a nice overview of the issue.</div>
<div class="p1">
<br />
The central idea is that it is now possible to use HTML, CSS, and JavaScript to build entire web applications, including interacting with the datastore.</div>
<div class="p1">
<br />
OK. We get it. What about specifics?</div>
<div class="p1">
<br />
A little more Googling, leads me, as with most questions about programming, to stackoverflow...</div>
<div class="p3">
<span class="s2"><a href="http://stackoverflow.com/questions/4180682/are-there-any-full-stack-server-side-javascript-frameworks-available">http://stackoverflow.com/questions/4180682/are-there-any-full-stack-server-side-javascript-frameworks-available</a></span></div>
<div class="p1">
<br />
That’s very helpful, but it’s interesting to note that that post is from almost two years ago.</div>
<div class="p1">
<br />
To be frank, I wasn’t blown away by the response I got. There didn’t seem to be a “clear winner” in the race to become the default solution. The playing field seems a few players short of a full squad.</div>
<div class="p1">
<br />
Nonetheless, a few names did keep cropping up.</div>
<div class="p1">
<br />
I shall endeavor to list the major players here (please let me know in the comments if there are any other things I should mention).</div>
<div class="p2">
<br /></div>
<div class="p3">
<span class="s1"><b>Derby</b>: <span class="s2"><a href="http://derbyjs.com/">http://derbyjs.com/</a></span></span></div>
<div class="p5">
<span class="s4">“</span>Derby is built on top of popular libraries, including <a href="http://nodejs.org/"><span class="s5">Node.js</span></a>, <a href="http://expressjs.com/"><span class="s5">Express</span></a>, <a href="http://socket.io/"><span class="s5">Socket.IO</span></a>, <a href="https://github.com/substack/node-browserify"><span class="s5">Browserify</span></a>, <a href="http://learnboost.github.com/stylus/docs/iteration.html"><span class="s5">Stylus</span></a>, <a href="http://lesscss.org/"><span class="s5">LESS</span></a>,<a href="https://github.com/mishoo/UglifyJS"><span class="s5">UglifyJS</span></a>, <a href="http://www.mongodb.org/"><span class="s5">MongoDB</span></a>, and soon other popular databases and datastores. These libraries can also be used directly. The data synchronization layer, <a href="http://racerjs.com/"><span class="s5">Racer</span></a>, can be used separately. Other client libraries, such as jQuery, and other Node.js modules from npm work just as well along with Derby.”</div>
<div class="p1">
Derby’s docs also includes a handy section called “Why Not Use Rails and Backbone?” that supports the central thesis of this post.</div>
<div class="p2">
<br /></div>
<div class="p3">
<span class="s1"><b>Tower.js</b>: <span class="s2"><a href="http://towerjs.org/">http://towerjs.org/</a></span></span></div>
<div class="p6">
<span class="s4">“</span>Default Development Stack</div>
<ul class="ul1">
<li class="li7">MongoDB (database)</li>
<li class="li7">Redis (background jobs)</li>
<li class="li7">CoffeeScript</li>
<li class="li7">Stylus</li>
<li class="li7">Jasmine (tests)</li>
<li class="li7">jQuery“</li>
</ul>
<div class="p3">
<span class="s1"><b><br />
</b></span> <span class="s1"><b>Ringo</b> (Helma): <a href="http://ringojs.org/"><span class="s2">http://ringojs.org/</span></a> <span class="s2"><a href="http://www.helma.org/">http://www.helma.org/</a></span></span></div>
<div class="p1">
I’m struggling a bit to figure out exactly how these projects are related to one another (Ringo appears to be a newer implementation of Helma, but I’m not quite sure if it’s meant to supercede it). In any case, Ringo and Helma seem to be the big players based on Rhino (therefore Java) and use JDBC for database connectivity, as well as Jetty.</div>
<div class="p2">
<br /></div>
<div class="p1">
So, my mission, should I decide to accept it, is to attempt to evaluate this stuff over the next few weeks and post anything interesting that I find. In particular, which, if any, of these solutions seems Ready for Prime Time.</div>
<div class="p1">
<br />
If I can bring myself to do it, I may re-code the above referenced Terrorism database application using one of these stacks. Hopefully, it will be more fun than the first time!</div>
<div class="p2">
<br /></div>Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com4tag:blogger.com,1999:blog-4261954431421179908.post-91710242461245611902011-02-06T19:26:00.005+00:002011-02-07T16:43:01.875+00:00Fun with Financial Data (or As Much Fun As You Can Have Without Actually Making Any Money)I've started reading the book <a href="http://www.amazon.com/Data-Analysis-Open-Source-Tools/dp/0596802358">Data Analysis with Open Source Tools</a> by Philipp K. Janert. It's very well written and informative, and I highly recommend it. I've only scratched the surface so far, but I decided that I wanted to try to apply some ideas from it to a problem that had been kicking around in my head for while.<br />
<br />
And then I figured: why not write a blog post about it! (After all, no one reads this thing anyway ;-)<br />
<br />
This problem occurred to me a while back while reading another O'Reilly book: <a href="http://www.amazon.com/Programming-Collective-Intelligence-Building-Applications/dp/0596529325">Programming Collective Intelligence</a> by Toby Segaran. The idea involves looking at a couple of pieces of information that are available on the web: Financial price data and financial news (both of which are freely available at <a href="http://developer.yahoo.com/finance/">Yahoo! Finance</a>). I realize that the data that Yahoo publishes is not particularly real-time and one wouldn't probably want to build an HFT system on it, but it will serve to do a little studying.<br />
<br />
I didn't really have a hypothesis in mind when I started. Taking a cue from Mr. Janert, I decided to take a look at the data with an open mind (no pre-conceived notions). At the heart of this inquiry, however, is a two-tier question:<br />
<ol><li>Is there some kind of correlation between when news gets published online and when stock prices move? and</li>
<li>Is there a predictable sequential relationship that can be construed?</li>
</ol>Why do this? Well, it's interesting. I'd been interested for a while in finding a worthwhile excuse to do something with financial data. Besides, there's a small chance that it might reveal an interesting opportunity (if news precedes predictable changes in price? Cha-ching!). More likely, however, I was expecting that I'd find that We, The Sheeple are trailing the HFT systems and insiders and there is little or no hope for us little folks to make a thin dime playing the market.<br />
<br />
Please note that this is a first pass at this. By no means am I imagining that this is a comprehensive approach. Indeed, I would be very grateful if anyone has any ideas for what to look at next (please comment!).<br />
<br />
For those who like the punchline first, here's the spoiler: the evidence is inconclusive as to whether any predictions can be made in price change based on headlines. Further study might reveal something, but at first glance, it just doesn't add up (pun intended).<br />
<br />
Conversely, and somewhat surprisingly, it is not conclusive that price changes precede news. In other words, it is not clear cut that the news follows the market.<br />
<br />
However, there is clearly a correlation in the data: big price shifts, high volume, and an increase in published news tend to go hand-in-hand. This is an unremarkable finding, but nonetheless, it was an interesting exercise.<br />
<br />
Now, despite having warned you that these findings are dull as dishwater, if you are really interested, I presume you are still with me...<br />
<br />
To analyze the problem, the first thing I did was to set up a method for capturing data. Once I had a bit of data to look at, then I could begin my inquiry.<br />
<br />
I don't really want to spend a lot of time discussing the specific technologies employed, as that's not really the point of this post, but I will just summarize this step briefly:<br />
<ol><li>A couple of shell scripts employing curl (via cron) to grab data at reasonable intervals: every minute for price data, and every hour for news data.</li>
<li>A couple of Scala scripts for parsing that data and dumping into a sqlite database.</li>
<li>SQL queries to grab the data (grouped as appropriate) in CSV format to pull into Excel (where I can create charts, where I can create charts for visualization, etc.)</li>
</ol>As you can see, very low-tech, indeed. All in the spirit of keeping it simple. I thought about using <a href="http://matplotlib.sourceforge.net/">matplotlib</a>, which is a great Python-based tool for doing data analysis that I've had the pleasure of using a number of times in the past. But, as the data set I was focusing on was pretty small, Excel did the job nicely.<br />
<br />
In another iteration of this inquiry, I might be inclined to try out <a href="http://www.r-project.org/">R</a> or <a href="http://www.gnu.org/software/octave/">Octave</a>, and dig a little deeper. Perhaps I'll even take a further look at what data is available and try to expand the reach. If I'm feeling especially crazy, I might even give <a href="http://hadoop.apache.org/">Hadoop</a> a try.<br />
<br />
For purposes of this exercise, I will be looking at this data over the course of a single trading week: January 10-14 2011 (inclusive), and I'm just grabbing data on stocks that make up the Dow Jones Industrial Average. No particular reason other than that that was a convenient week to do it, and it seemed like a sufficient amount of time for this initial undertaking.<br />
<br />
I figured I would take a "big picture" view first. See whether just glancing at the data indicates anything interesting. If something jumps out, then drill down further and see where it takes me.<br />
<br />
To put this in context, here's a graph showing the Dow's movement around that time:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixS_p_wlMmJqFzNTY-hqa6jOSreQEWs2_DoBPvJqfQGPfKImvK6wLUxiLyTrrJe0z-lNGphL3LZlTJQRDaTLso1jQhZbEpyuRD_IqXHrBbD8w22iEdA-WpCWa8f7wEW0mn9LKeLC1zedJl/s1600/DowJones-Jan10-14.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixS_p_wlMmJqFzNTY-hqa6jOSreQEWs2_DoBPvJqfQGPfKImvK6wLUxiLyTrrJe0z-lNGphL3LZlTJQRDaTLso1jQhZbEpyuRD_IqXHrBbD8w22iEdA-WpCWa8f7wEW0mn9LKeLC1zedJl/s400/DowJones-Jan10-14.png" width="400" /></a></div><br />
Prices had been coming down the previous week, but were poised to generally go up this week (hindsight being 20/20 and all).<br />
<br />
The first things I want to look at are:<br />
<ol><li>How many news headlines are there per stock per day? (And look for spikes in these numbers.)</li>
<li>Significant shifts in prices.</li>
<li>Significant shifts in volumes.</li>
</ol><br />
That alone should establish where there's a relationship that's worth exploring. Doing this on a per stock per day over these few days should be sufficient to start. Then, if motivated, look at the intra-day prices and specific publish times of articles to see if one can determine whether it's the chicken or the egg that comes first.<br />
<br />
To start, here's some summary information about our headline data per stock per day (biggest ranges, and largest gaps between mean and max are shown highlighted):<br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSDfMsny5Rc0Zuqwd_K2AfDZLUAKO5zkClFPQQsKg7IrW1_lzdvG6JeDwyzoSfXWjtiKb_TpHICbwQdEMSbJlFD5sNHlRQ2fpJriH6nTmah0LavwfVMNHcJTWvgYWJwHjqKpP1j8w3zmSF/s1600/data-headlines1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="197" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSDfMsny5Rc0Zuqwd_K2AfDZLUAKO5zkClFPQQsKg7IrW1_lzdvG6JeDwyzoSfXWjtiKb_TpHICbwQdEMSbJlFD5sNHlRQ2fpJriH6nTmah0LavwfVMNHcJTWvgYWJwHjqKpP1j8w3zmSF/s400/data-headlines1.png" width="400" /></a></div><br />
<br />
And here it is again, measuring number of standard deviations from the mean (largest numbers highlighted):<br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuOSxgZnNEK8SbrsI2tkDKgnTJQpetceIh6A3CjqmVYZXsfLhj6ZjkT9rjmmwaKZOwuKV3YfXS9cFUceajPS9Qfwq-N4A-GQxQwlj53ChQC7psoWA16sVeVoe0J8JUCFEPcvgdHV9lSbe-/s1600/data-headlines2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuOSxgZnNEK8SbrsI2tkDKgnTJQpetceIh6A3CjqmVYZXsfLhj6ZjkT9rjmmwaKZOwuKV3YfXS9cFUceajPS9Qfwq-N4A-GQxQwlj53ChQC7psoWA16sVeVoe0J8JUCFEPcvgdHV9lSbe-/s400/data-headlines2.png" width="400" /></a></div><br />
<br />
Those spikes are a lot clearer when looking at them charted:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0KxwSH_FQy8H6DpmA1Zw_RtyoGojMjgXoHQeA0_jqJ0c7ckfJpIEcnEywmv62uos9YS2l1fsod7pSdWKopdKX-LIoWq3-ZfzJhigFCHc9STrXyeVfMG776Pm9rCFI78Xvmi1Viwbvtz6Q/s1600/HeadlinesByCount.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0KxwSH_FQy8H6DpmA1Zw_RtyoGojMjgXoHQeA0_jqJ0c7ckfJpIEcnEywmv62uos9YS2l1fsod7pSdWKopdKX-LIoWq3-ZfzJhigFCHc9STrXyeVfMG776Pm9rCFI78Xvmi1Viwbvtz6Q/s400/HeadlinesByCount.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhMApKMaWKW-UGiaEYYNvbUxlskNjP2gBLrpJLPwfVBK7UZsXicrMsrxXA0OhYyqEIqn_YA3b3FiAw0Bwu8HEqPgyT9Hx4W4MA30udZh_rqnhQzeuqGHdtOcD3r6WIopy8aG24UgSNUJMR/s1600/HeadlinesByStdDev.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhMApKMaWKW-UGiaEYYNvbUxlskNjP2gBLrpJLPwfVBK7UZsXicrMsrxXA0OhYyqEIqn_YA3b3FiAw0Bwu8HEqPgyT9Hx4W4MA30udZh_rqnhQzeuqGHdtOcD3r6WIopy8aG24UgSNUJMR/s400/HeadlinesByStdDev.png" width="400" /></a></div><br />
Looking at these, just based on spikes in number of news articles, I'd be interested in looking further at the following:<br />
<br />
(Because I know what headlines at Yahoo tend to get grouped with stocks in the same industries, I'm looking for those relationships as well.)<br />
<br />
<ol><li>DuPont (DD), Johnson & Johnson (JNJ), and Procter & Gamble (PG) on Monday.</li>
<li>AT&T (T) and Verizon (VZ) on Tuesday.</li>
<li>Merck (MRK) and Pfizer (PFE) on Thursday.</li>
<li>Bank of America (BAC) and JP Morgan (JPM) on Friday.</li>
</ol><br />
Now, let's see if that jives with what we can see in price data (note: this is shown by price range for the day as a fluctuation (max-min) percentage from the previous day's close):<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRxIHm3cB6waUnZvxILo6UlukUWGvuIrEaqAsDYCXIbo4J6Fm8QJQmX3huKY0d1-nEa0QAd1mPkHFb-ixtf_-KvLEN6wPW29OMCdrEhmkq8VEjXeeycDw5mRqBALPUzOM2o1H1FSN28aAO/s1600/data-prices1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="291" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRxIHm3cB6waUnZvxILo6UlukUWGvuIrEaqAsDYCXIbo4J6Fm8QJQmX3huKY0d1-nEa0QAd1mPkHFb-ixtf_-KvLEN6wPW29OMCdrEhmkq8VEjXeeycDw5mRqBALPUzOM2o1H1FSN28aAO/s400/data-prices1.png" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ7Zw_8ejUKcxCnM4hXlNO_8y-QzwBSRPuE7d-URoihPk6lUecqU5ccc_SrkaD7tmzZqDsbjLNLeJJyLNuvjNA2JXk9E7EZUm9fVyiLrrHLo_i9FZm87R2dDz9LBp8x4xG7ZUGn_KAQHQB/s1600/PriceChangePct.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ7Zw_8ejUKcxCnM4hXlNO_8y-QzwBSRPuE7d-URoihPk6lUecqU5ccc_SrkaD7tmzZqDsbjLNLeJJyLNuvjNA2JXk9E7EZUm9fVyiLrrHLo_i9FZm87R2dDz9LBp8x4xG7ZUGn_KAQHQB/s400/PriceChangePct.png" width="400" /></a></div><br />
Looking at the price data, there is certainly enough coming through to believe that there is a correlation here. Much of the data points to the same stocks/days. I might be inclined to filter my list, however, as it least some of the headline data is likely to be associated with certain stocks only because of the industry relationship:<br />
<br />
<ol><li>DuPont (DD) on Monday.</li>
<li>AT&T (T) and Verizon (VZ) on Tuesday.</li>
<li>Merck (MRK) on Thursday.</li>
<li>Bank of America (BAC) and JP Morgan (JPM) on Friday.</li>
</ol><br />
That reduces it to six different stocks on four different days, which seems like a reasonable number to analyze further (with two industry relationships to look at).<br />
<br />
I might also be inclined to add some based on the price data alone, like McDonald's (MCD) or American Express (AXP) on Friday, but when looking at the headline data, there's not really enough to go on. (Most of the headlines for McDonald's on the Friday were actually about Starbucks.)<br />
<br />
And, just to complete the picture, I'll have a look at volumes as well.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAuCHnR1Lm-LQ0fMV42xrJ_MsoVsiwKk396APifSVH1V_5ntq99-4hiKe18v6vkIGfyAB9DEu7qBldqjNoy2P8NH2VoWADDMcLbwOEUwwRpPhFnbf6HkNQulkQk_PVOlCot38tyt__q96s/s1600/data-volume1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAuCHnR1Lm-LQ0fMV42xrJ_MsoVsiwKk396APifSVH1V_5ntq99-4hiKe18v6vkIGfyAB9DEu7qBldqjNoy2P8NH2VoWADDMcLbwOEUwwRpPhFnbf6HkNQulkQk_PVOlCot38tyt__q96s/s400/data-volume1.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9H_oyhXaJ37t_4fvk5lgWeyPRwMEk60CN9v-TmlLNvyjWaWrvOhkHNfZX52JgyWtILOFPObDmHVbWu09n0XR5U_ftWMVeYpsLpVDUZje14wwr18AYpat_hhwyf6_FUxOm5L7JDNpAaBNm/s1600/data-volume2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="341" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9H_oyhXaJ37t_4fvk5lgWeyPRwMEk60CN9v-TmlLNvyjWaWrvOhkHNfZX52JgyWtILOFPObDmHVbWu09n0XR5U_ftWMVeYpsLpVDUZje14wwr18AYpat_hhwyf6_FUxOm5L7JDNpAaBNm/s400/data-volume2.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSDezRjxOyRlJPAPLwwr2Nn5pqDk5kaD-OwfUOclGTiaS546mxOKWrvO53H7nR2ItJs22xUKRhigh3CEWvZiCtNkmUu9eZdsLHsrJOCZJwxk410tt8ErLypKuwSX4zaPMyiV_CpjCpxZaq/s1600/VolumeSpikes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSDezRjxOyRlJPAPLwwr2Nn5pqDk5kaD-OwfUOclGTiaS546mxOKWrvO53H7nR2ItJs22xUKRhigh3CEWvZiCtNkmUu9eZdsLHsrJOCZJwxk410tt8ErLypKuwSX4zaPMyiV_CpjCpxZaq/s400/VolumeSpikes.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHYxpYrg6ErAyV0_5q4u8Vzl6MsIl4KE5USedkuz24XFGj4X7sOyKOM8CSwK5ejO7mX-ihaUeaETmPJzPNrkj1av9gzZJS7HLFklSewhk8bMl21oOtHPJSn25t2na6i3CnpEHU6Ye4hsOH/s1600/VolChangeByStdDev.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHYxpYrg6ErAyV0_5q4u8Vzl6MsIl4KE5USedkuz24XFGj4X7sOyKOM8CSwK5ejO7mX-ihaUeaETmPJzPNrkj1av9gzZJS7HLFklSewhk8bMl21oOtHPJSn25t2na6i3CnpEHU6Ye4hsOH/s400/VolChangeByStdDev.png" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"></div><br />
Again, a lot of the same players emerge, which leads me to think that I'm on the right track. The correlation is clear. Now, I'll have to look at the intra-day data to see if one can draw any conclusions moment-to-moment on the specific scenarios that I've identified.<br />
<br />
<h4>DuPont (DD): Monday January 10th, 2011</h4><h4><span style="font-weight: normal;">First off, the data... (Note that the price column is price range during that hour as a percentage)</span></h4><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYm_fSyLh1KJZzqFxEThUtMY_fwIWrERodG4HVrCmDSn4V15odcAuc0z66k0er9sB3KX3QWY1_EEMS5UlIReqe8GCXSKUrA_GY2Qf71aY9e_7hv4fk5ZcJii3IyKJ6wPp-kmCruS3fCS46/s1600/dd-data.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="185" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYm_fSyLh1KJZzqFxEThUtMY_fwIWrERodG4HVrCmDSn4V15odcAuc0z66k0er9sB3KX3QWY1_EEMS5UlIReqe8GCXSKUrA_GY2Qf71aY9e_7hv4fk5ZcJii3IyKJ6wPp-kmCruS3fCS46/s400/dd-data.png" width="400" /> </a></div><div class="separator" style="clear: both; text-align: center;"><br />
</div>The following chart shows the deltas in headline count, price, and volume as a percentage of the total change over the course of the day (this allows unlike numbers to be compared in like manner). In particular, note that I've summed the delta of pre-open headlines into the first hour's number, which might exaggerate it's effect on the chart. But looking at the data you can certainly see that there was significant activity pre-open.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgx6Gw8D0U1cEUG8kQ_OZyIb6keABBYZEGwl5HQrlKU8Q0oWDTYvWjp9qJGA2DQVbHC6BYsIMYPRSYyldCosEXqSKx9YlE2c0pqZycgkZCrwNdZQQ1GqjJocPj_P3ihbyCPJT_wFGAk1dO8/s1600/dd-deltas.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgx6Gw8D0U1cEUG8kQ_OZyIb6keABBYZEGwl5HQrlKU8Q0oWDTYvWjp9qJGA2DQVbHC6BYsIMYPRSYyldCosEXqSKx9YlE2c0pqZycgkZCrwNdZQQ1GqjJocPj_P3ihbyCPJT_wFGAk1dO8/s400/dd-deltas.png" width="400" /></a></div><br />
<br />
And, price for the day...<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2NT0Im6aJ7UgNSZNS9pnU51Y9qzUSDovGgEK7SA5d-LU2zYrwoLrEGW2-TkWi2d8G_h4mSJBu1rWgJC1s-xja6hxzdijIkdxbVi3lrQk34eouTtdt_vFLe6fMptrnjEQshGmIEv5ezjPR/s1600/dd-price.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2NT0Im6aJ7UgNSZNS9pnU51Y9qzUSDovGgEK7SA5d-LU2zYrwoLrEGW2-TkWi2d8G_h4mSJBu1rWgJC1s-xja6hxzdijIkdxbVi3lrQk34eouTtdt_vFLe6fMptrnjEQshGmIEv5ezjPR/s400/dd-price.png" width="400" /></a></div><br />
<br />
One thing's for certain, there was definitely a spike in the number of headlines at 11 am which <i>followed</i> the steep change in price. But most of the headlines from that day were before the bell, and the big dip in price happened before 10 am. By 11 am the price was fairly stable.<br />
<br />
So, what were the headlines? Well, here are all the headlines that were published before 11 am:<br />
<ul><li>07:07 "Duke, Progress Energy, Sara Lee, Strayer, WellPoint: U.S. Equity Preview"</li>
<li>07:15 "US STOCKS-Futures lower on Portugal concerns despite M&A"</li>
<li>07:24 "CNNMoney Pre-Market Report - Jan. 10, 2011"</li>
<li>07:25 "DuPont to Buy Danisco for $5.8 Billion"</li>
<li>07:25 "Futures Fall; DuPont, Earnings in Focus"</li>
<li>07:32 "Disappointing Jobs Data Weighs on Futures"</li>
<li>07:45 "[$$] Danisco to Recommend DuPont Bid"</li>
<li>07:56 "US STOCKS-Futures lower as Portugal concerns offset M&A"</li>
<li>08:08 "Let's Make a Deal"</li>
<li>08:19 "Futures follow Asia, Europe lower"</li>
<li>08:27 "UPDATE 1-Goldman Sachs raises US chemicals sector to attractive"</li>
<li>08:30 "[$$] The Day Ahead: Merger Monday"</li>
<li>08:40 "UPDATE 3-DuPont's Danisco bid sends sector shares higher"</li>
<li>08:53 "Before the Bell: Duke Energy, DuPont, Verizon in spotlight"</li>
<li>08:59 "U.S. Stocks Poised For Weak Open"</li>
<li>09:09 "[video] News Hub: Before the Opening Bell [1.4 min]"</li>
<li>09:14 "[video] Faber Report"</li>
<li>09:20 "Danish currency firms on Danisco takeover"</li>
<li>09:40 "CNNMoney Market Report - Jan. 10, 2011"</li>
<li>09:44 "Dow Falls at Open Despite Rise in M&A"</li>
<li>09:47 "Verizon iPhone Announcement Seen for Tuesday"</li>
<li>09:48 "Stocks Dip on Euro Debt Concerns"</li>
<li>09:50 "Portugal Bailout Push Mars Merger Monday"</li>
<li>09:59 "[audio] Opening Bell from MarketWatch Radio Network [1.0 min]"</li>
<li>10:00 "Monday Morning Quarterback: Mergers, Foreclosure and Hank Moody"</li>
<li>10:16 "Wall St down as Portugal woes offset M&A deals"</li>
<li>10:39 "US equities fall on eurozone fears"</li>
<li>10:57 "Stocks to Watch: Union Pacific, Exxon and More ..."</li>
</ul>Clearly the buyout of Danisco was a big factor here, but I'm having a hard time understanding why it had the effect it did on price. It almost seems counter-intuitive that the share price should fall so dramatically in the first 20 minutes of trading, given the seemingly "good news" of this buyout. Perhaps it's something to do with exchange rates and the European problems that were weighing on the market? Was it this: <a href="http://www.reuters.com/article/2011/01/10/denmark-crown-danisco-idUSDKT00521320110110?feedType=RSS&feedName=rbssFinancialServicesAndRealEstateNews&rpc=43">Danish currency firms on Danisco takeover</a>? (09:20)<br />
<br />
Interesting, but by no means conclusive.<br />
<br />
One headline in there jumps out, however, and makes a nice segue:<br />
<br />
09:47 "Verizon iPhone Announcement Seen for Tuesday"<br />
<br />
That'll explain our next subject: Verizon and AT&T activity on Tuesday.<br />
<br />
<h4> Verizon (VZ): Tuesday January 11th, 2011</h4><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgik-9T_DEyqxMX320KOvpZIIs-u1yleMkDDSkhNOb1MOrq5xY7DeapF5B7Zjv9hVuLPPYrx04tG4XF6LzdhC6zxxeDXUS23pk7_NxirUYWg7JZRAOtdtQhmWyn8NT2VzEoVPXxqZ1gJMzz/s1600/vz-data.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="258" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgik-9T_DEyqxMX320KOvpZIIs-u1yleMkDDSkhNOb1MOrq5xY7DeapF5B7Zjv9hVuLPPYrx04tG4XF6LzdhC6zxxeDXUS23pk7_NxirUYWg7JZRAOtdtQhmWyn8NT2VzEoVPXxqZ1gJMzz/s400/vz-data.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHCTc9pfI3RFP6Jcqy34d0-zNjYNhwcwmZD_B-aIPr0b1VyZHMb14rWB2_v-9ZAvgrWmXtDcrUpLidHYKf_KGAqKMxgQDki87ICmagc4qSIVHV7RefWOQ0bBUo9tgoeaEtixDB-BcVrWYn/s1600/vz-deltas.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHCTc9pfI3RFP6Jcqy34d0-zNjYNhwcwmZD_B-aIPr0b1VyZHMb14rWB2_v-9ZAvgrWmXtDcrUpLidHYKf_KGAqKMxgQDki87ICmagc4qSIVHV7RefWOQ0bBUo9tgoeaEtixDB-BcVrWYn/s400/vz-deltas.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH4-qPja8EHsvRt7CZ19qx-EOG5T5Xl_AtJRZYabLh4lfxFukiQpgpsq9OCi2VcXgX0XVYwcHAksUMYnNKd-vcD73C-mk8cRujE7DI_ok5NJTnP9gqARWinqGCBL8LjM1NMKfKkh994DYt/s1600/vz-price.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH4-qPja8EHsvRt7CZ19qx-EOG5T5Xl_AtJRZYabLh4lfxFukiQpgpsq9OCi2VcXgX0XVYwcHAksUMYnNKd-vcD73C-mk8cRujE7DI_ok5NJTnP9gqARWinqGCBL8LjM1NMKfKkh994DYt/s400/vz-price.png" width="400" /></a></div><h4 style="font-weight: normal;"> The spike in price at around 11 am when all those articles were published (which corresponded to the time of the Verizon iPhone launch) is certainly interesting. But I don't think it would take a genius to figure out that that might happen.</h4>The other scenarios below also confirm what was stated previously, so I'm happy to leave the rest of this as an exercise for the reader. There is a correlation between headline count and price changes, but not a clearly sequential one. Again, I welcome feedback or ideas. I hope to come back to this at some stage and delve a little deeper. In the meantime, here's the rest of that data, starting with the Verizon headlines...<br />
<ul><li>00:01 "IPhone May Cost Verizon Wireless $5 Billion in Subsidies in First Year"</li>
<li>00:15 "ANALYSIS - AT&T faces tough year after losing iPhone exclusive"</li>
<li>02:47 "Telecoms' Dividend Yields"</li>
<li>03:03 "With iPhone Taunts, AT&T-Verizon Rivalry Escalates"</li>
<li>03:51 "11 High Yield Stocks to Consider Now"</li>
<li>05:00 "[video] Verizon iPhone Data Plan"</li>
<li>06:00 "Nielsen: Young People Across The Globe Love Their Cell Phones (But Use Them Differently)"</li>
<li>06:45 "Half of Dow Companies to Post Profit Gains"</li>
<li>06:50 "Ahead of the Bell: Verizon iPhone"</li>
<li>07:27 "10 Things You Need To Know Before The Opening Bell"</li>
<li>07:27 "10 Things You Need To Know Before The Opening Bell"</li>
<li>07:30 "CNNMoney Pre-Market Report - Jan. 11, 2011"</li>
<li>07:32 "Lowenstein Says Verizon's Network Ready to Handle IPhone: Video"</li>
<li>07:39 "10 Things You Need To Know This Morning"</li>
<li>07:40 "iPhone Raises Stakes of AT&T-Verizon Rivalry"</li>
<li>07:45 "Verizon big winner from having iPhone? Not so fast"</li>
<li>07:58 "AT&T and Verizon Trade Taunts Over iPhone"</li>
<li>08:03 "[video] 3 Stocks I Saw on TV"</li>
<li>08:22 "Will Verizon Demolish AT&T? It Did So Long Ago"</li>
<li>08:24 "The Big Questions About Verizon's IPhone"</li>
<li>08:24 "Verizon big winner from having iPhone? Not so fast"</li>
<li>08:25 "McCourt Says Verizon's Network Can Handle Apple's IPhone: Video"</li>
<li>08:36 "How Important Is Offering the iPhone on Verizon to Apple?"</li>
<li>08:45 "[$$] ETF Play of the Day: XLK"</li>
<li>08:56 "Liveblog: Verizon set to launch the iPhone. Finally."</li>
<li>09:02 "[video] AM Report: Chinese Stealth Jet Takes Flight [8.0 min]"</li>
<li>09:17 "[video] 3 Networking Stocks Set for Cloud Boost"</li>
<li>09:29 "Verizon iPhone Faces Even Bigger Foe"</li>
<li>09:33 "Trujillo Says 'Many Questions' Remain For Verizon IPhone: Video"</li>
<li>09:45 "Verizon IPhone Event Live Blog"</li>
<li>09:45 "[video] News Hub: Verizon iPhone: How will AT&T Respond? [2.7 min]"</li>
<li>09:52 "Verizon's Big iPhone Unveil"</li>
<li>09:52 "[video] Verizon's iPhone Buzz"</li>
<li>09:54 "Mobile Broadband Subs Seen Hitting 1 Billion In 2011"</li>
<li>09:58 "[audio] Opening Bell from MarketWatch Radio Network [1.0 min]"</li>
<li>10:03 "Video: Is the Verizon iPhone a Threat to Android?"</li>
<li>10:05 "Verizon iPhone expected to have unlimited data plan"</li>
<li>10:14 "[video] iPhone or Not, Verizon Is A Winner"</li>
<li>10:19 "Verizon iPhone: Live Blog"</li>
<li>10:37 "Verizon: Stay Tuned"</li>
<li>10:47 "LIVE: The Verizon iPhone Event"</li>
<li>10:53 "Faber Report: What Will Drive Verizon?"</li>
<li>10:54 "LIVE FROM NEW YORK: Verizon Gets the iPhone"</li>
<li>10:54 "Verizon iPhone: Will It Fix Verizon's $100 Billion Conundrum?"</li>
<li>10:55 "Live Blog: Verizon's iPhone Announcement"</li>
<li>10:57 "Live Blogging the Verizon iPhone Announcement"</li>
<li>11:18 "Verizon to launch iPhone 4 in February"</li>
<li>11:20 "Verizon iPhone: The basics"</li>
<li>11:24 "The Biggest Surprise About the Verizon iPhone: It's a Mobile Hotspot"</li>
<li>11:24 "Verizon to Carry Apple iPhone, Ending Months of Anticipation"</li>
<li>11:30 "Verizon Will Be Selling The iPhone Starting February 10th"</li>
<li>11:36 "Live Blog: Verizon's iPhone Announcement"</li>
<li>11:38 "UPDATE 2-Verizon Wireless to sell iPhone in February"</li>
<li>11:39 "PHOTOS: Verizon iPhone Has Different (Better?) Antenna Design Than AT&T iPhone"</li>
<li>11:39 "Verizon, AT&T, And Apple: Guess Which One's Stock Isn't Tanking Today"</li>
<li>11:41 "Verizon to Sell iPhone 4 on Feb. 10"</li>
<li>11:43 "Verizon Wireless to Offer iPhone 4 in February"</li>
<li>11:45 "Verizon iPhone: 'It Begins'"</li>
<li>11:50 "Verizon to begin selling iPhone 4 in February"</li>
<li>11:51 "Bizarre Fact of the Day: Tyler Durden and Louis Winthorpe III Are Practically the Same Person"</li>
<li>11:51 "Verizon to sell Apple iPhone"</li>
<li>11:51 "Verizon: Began Discussions With Apple In 2008; Tim Cook Talks"</li>
<li>11:54 "Verizon Wireless to sell iPhone in February"</li>
<li>12:17 "AT&T Preps for Verizon iPhone With New Ads, Price Cuts"</li>
<li>12:18 "[audio] Enderle: Verizon and Apple no match made in heaven [5.1 min]"</li>
<li>12:19 "Apple's Phil Schiller on Building the Verizon iPhone"</li>
<li>12:20 "AT&T Shrugs Off Verizon iPhone"</li>
<li>12:20 "COMPARISON: Verizon Vs. AT&T iPhone"</li>
<li>12:26 "FACTBOX - AT&T and Verizon wage battle over iPhone"</li>
<li>12:28 "Verizon iPhone: Yep, The Real Target is Google."</li>
<li>12:28 "Why No 4G iPhone?"</li>
<li>12:30 "AT&T: Five Reasons the Sky Won't Fall When iPhone Goes to Verizon"</li>
<li>12:35 "[audio] Midday Update from MarketWatch Radio Network [1.0 min]"</li>
<li>12:37 "FACTBOX - AT&T and Verizon wage battle over iPhone"</li>
<li>12:37 "Verizon Wireless to sell iPhone in February"</li>
<li>12:43 "Verizon to start selling iPhone on Feb. 10"</li>
<li>12:48 "Corrected: Factbox: AT&T and Verizon wage battle over iPhone"</li>
<li>12:50 "Verizon Will Offer the iPhone Next Month"</li>
<li>12:57 "Corrections: Verizon to begin selling iPhone 4 in February"</li>
<li>12:59 "Verizon Wireless to sell iPhone in February"</li>
<li>13:17 "Stronger earnings reports push stocks higher"</li>
<li>13:18 "Most active New York Stock Exchange-traded stocks"</li>
<li>13:18 "[video] Apple's iPhone 4 Comes to Verizon [1.4 min]"</li>
<li>13:19 "iPhone Will Bring 'Value' to Verizon Shareholders: CEO"</li>
<li>13:22 "Apple/Verizon Pt. 2: Should I stay or should I go now?"</li>
<li>13:23 "FACTBOX: AT&T and Verizon wage battle over iPhone"</li>
<li>13:23 "Halftime: Verizon iPhone To Drag Down Verizon Stock 10%?"</li>
<li>13:25 "[$$] iPhone Disappears Over the Verizon"</li>
<li>13:28 "[audio] Time for a tech detox? [1.9 min]"</li>
<li>13:29 "Wireless Struggles Against FCC Over Possible ""Bill Shock"" Rules"</li>
<li>13:34 "Verizon big winner from having iPhone? Not so fast"</li>
<li>13:38 "Verizon President Talks iPhone Strategy"</li>
<li>13:43 "FACTBOX: AT&T and Verizon wage battle over iPhone"</li>
<li>13:43 "US stocks up on strong earnings [at Financial times]"</li>
<li>13:45 "Verizon Gets the iPhone"</li>
<li>13:45 "Verizon to start selling iPhone on Feb. 10"</li>
<li>13:49 "Apple: No LTE iPhone This Year?"</li>
<li>13:53 "AT&T: We're ""Evaluating"" The New iPhone Mobile Hotspot Feature"</li>
<li>13:58 "Apple: Surprise! Street Expresses Delight With VZ iPhone"</li>
<li>14:00 "[$$] Catching the Wave: Euro Rally Due"</li>
<li>14:02 "Did Apple disappoint with no 4G iPhone?"</li>
<li>14:03 "Mobile Payments: What Verizon's iPhone Means for Square"</li>
<li>14:04 "Verizon to Sell Apple IPad That Connects Directly to Its Network"</li>
<li>14:09 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:10 "UPDATE 4-Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:13 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:25 "[audio] Before the Close from MarketWatch Radio Network [1.0 min]"</li>
<li>14:27 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:27 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:33 "Market Snapshot: U.S. stocks rise on earnings optimism"</li>
<li>14:33 "Verizon to start selling iPhone on Feb. 10"</li>
<li>14:41 "Analysts Go Out on Limb, Predict Verizon iPhone Will Be Big for Apple"</li>
<li>14:48 "Verizon Says You Can Switch If Santa Brought You a Droid"</li>
<li>14:49 "Verizon Wireless to Start IPhone Sales Next Month, Ending AT&T Exclusivity"</li>
<li>14:50 "Motorola: Opportunity In Lack Of LTE iPhone"</li>
<li>14:50 "Verizon iPhone, WoW: Hot Trends"</li>
<li>15:07 "Is Apple's iPhone a Scrooge?"</li>
<li>15:09 "How does the Verizon iPhone affect Android and AT&T?"</li>
<li>15:09 "More talk of smartphones diving into featurephone ground"</li>
<li>15:09 "Verizon iPhone unlimited data won't last forever"</li>
<li>15:09 "Why the Verizon 'dream phone' matters"</li>
<li>15:22 "[video] digits: Verizon Unveils Its iPhone [8.7 min]"</li>
<li>15:35 "With Verizon, Will Apple Take a Bite of More Enterprise?"</li>
<li>15:38 "5 'Dogs of the Dow' Worth Betting On"</li>
<li>15:38 "[video] iPhone: No wonders for Verizon stock"</li>
<li>15:44 "Why RadioShack Is Still Relevant"</li>
<li>15:50 "Apple: Jefferies Ups Tgt To $450; LTE iPhone In October?"</li>
<li>15:50 "BlackBerry PlayBook vs. Android Tablets"</li>
<li>15:51 "Happy Verizon iPhone Day!"</li>
<li>15:57 "[$$] Overheard"</li>
<li>15:58 "Verizon to start selling iPhone on Feb. 10"</li>
<li>16:12 "Verizon's Daniel Mead Doesn't Expect IPhone Price War: Video"</li>
<li>16:17 "Verizon to start selling iPhone on Feb. 10"</li>
<li>16:18 "Android to Lose From Verizon iPhone?"</li>
<li>16:20 "Investing for Income in 2011 with Stocks and Options Part 3: High-Yield Stocks"</li>
<li>16:24 "Tech Stocks: Techs post modest gains, but AMD sinks"</li>
<li>16:26 "[audio] Closing Bell from MarketWatch Radio Network [1.0 min]"</li>
<li>16:30 "Apple: Reasons For An LTE iPhone This Year"</li>
<li>16:30 "[video] digits: Verizon iPhone: What You Need to Know [4.1 min]"</li>
<li>16:30 "[video] digits:What Verizon iPhone Means for Apple, Google [2.1 min]"</li>
<li>16:31 "US STOCKS-Energy shares lift Wall St in light volume"</li>
<li>16:32 "Don't Buy The Verizon iPhone Yet, Unless You NEED A New Phone ASAP"</li>
<li>16:48 "[video] PM Report: China Stealth Fighter - Threat to U.S.? [10.2 min]"</li>
<li>16:50 "Stronger earnings reports push stocks higher"</li>
<li>17:50 "[video] Options Action"</li>
<li>17:54 "Stronger earnings reports push stocks higher"</li>
<li>17:55 "Factbox: AT&T and Verizon wage battle over iPhone"</li>
<li>18:09 "All Our iPhone Trades, Right Here & Now"</li>
<li>18:09 "Most active New York Stock Exchange-traded stocks"</li>
<li>18:44 "Doherty Sees 'Exodus' of AT&T IPhone Users to Verizon: Video"</li>
<li>19:00 "Lady Gaga Brags, U2 to Rival R.E.M. for 2011 Rock Crown: Preview"</li>
<li>19:40 "'Fast Money' Recap: Commodities Getting Frothy"</li>
<li>20:21 "MiFi Come, MiFi Go: Verizon Wireless To Sell iPads With 3G Connectivity"</li>
<li>22:02 "Tuesday ETF Roundup: UNG Surges on Weather, IYZ Tumbles on Increased Competition"</li>
<li>23:38 "Adding Up Parts at Goldman Sachs"</li>
</ul><br />
<h4>AT&T (T): Tuesday January 11th, 2011</h4><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDyLtLx6n7FpeYUF4v8xW5o4bAqMMl6ebCqEs5Cd5d-utMtkSd5oqDZAXHxqEIOeeRYekRKH5mvxz7bfeU5AdeWnXDTq0GasFBrdoaNtU5xOAIpIMGWKhRHcrbazRgeocRs_QjiutDr3m4/s1600/t-data.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDyLtLx6n7FpeYUF4v8xW5o4bAqMMl6ebCqEs5Cd5d-utMtkSd5oqDZAXHxqEIOeeRYekRKH5mvxz7bfeU5AdeWnXDTq0GasFBrdoaNtU5xOAIpIMGWKhRHcrbazRgeocRs_QjiutDr3m4/s400/t-data.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgM7gtxvOWYwbQvDEaiS-DUNr_x7rNH_rK56UNGiRImYVLEYqGweXe3d5RPkrdR8tjbNkid7gIAELPyIESzYj4cONv70uGIesR40wzAvwSEW1jBnJJp_aZr6Nnr3LxvhDWDKbLeFuLIjMlN/s1600/t-deltas.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgM7gtxvOWYwbQvDEaiS-DUNr_x7rNH_rK56UNGiRImYVLEYqGweXe3d5RPkrdR8tjbNkid7gIAELPyIESzYj4cONv70uGIesR40wzAvwSEW1jBnJJp_aZr6Nnr3LxvhDWDKbLeFuLIjMlN/s400/t-deltas.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbUW20v1UaAH370TZXkO4QD2SvfrF3xOsU-o530NTT1BIortpy3PW8ord256Awln_hLm9FxSWwFXhRsc3qaJSg0f55akAaIHLCWcaXDgnDieZVkcUuwT0hbqqPE8NGgiZO7gBc6fmQ6BMT/s1600/t-price.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbUW20v1UaAH370TZXkO4QD2SvfrF3xOsU-o530NTT1BIortpy3PW8ord256Awln_hLm9FxSWwFXhRsc3qaJSg0f55akAaIHLCWcaXDgnDieZVkcUuwT0hbqqPE8NGgiZO7gBc6fmQ6BMT/s400/t-price.png" width="400" /></a></div><br />
<ul><li>00:01 "IPhone May Cost Verizon Wireless $5 Billion in Subsidies in First Year"</li>
<li>00:15 "ANALYSIS - AT&T faces tough year after losing iPhone exclusive"</li>
<li>02:47 "Telecoms' Dividend Yields"</li>
<li>03:03 "With iPhone Taunts, AT&T-Verizon Rivalry Escalates"</li>
<li>05:00 "[video] Verizon iPhone Data Plan"</li>
<li>05:57 "Cramer's Lightning Round - Viva Las Vegas (1/10/11)"</li>
<li>06:00 "Nielsen: Young People Across The Globe Love Their Cell Phones (But Use Them Differently)"</li>
<li>06:21 "[$$] RadioShack Needs Verizon Wireless"</li>
<li>06:45 "Half of Dow Companies to Post Profit Gains"</li>
<li>06:50 "Ahead of the Bell: Verizon iPhone"</li>
<li>07:15 "GM returning to Super Bowl for first time since '08."</li>
<li>07:27 "10 Things You Need To Know Before The Opening Bell"</li>
<li>07:27 "10 Things You Need To Know Before The Opening Bell"</li>
<li>07:32 "Lowenstein Says Verizon's Network Ready to Handle IPhone: Video"</li>
<li>07:40 "iPhone Raises Stakes of AT&T-Verizon Rivalry"</li>
<li>07:41 "10 Things You Need to Know Before the Opening Bell"</li>
<li>07:45 "Verizon big winner from having iPhone? Not so fast"</li>
<li>07:58 "AT&T and Verizon Trade Taunts Over iPhone"</li>
<li>08:03 "[video] 3 Stocks I Saw on TV"</li>
<li>08:05 "AT&T-Verizon rivalry kicks up a notch"</li>
<li>08:09 "10 Unusual Stocks Attracting Huge Interest This Morning"</li>
<li>08:22 "Will Verizon Demolish AT&T? It Did So Long Ago"</li>
<li>08:24 "The Big Questions About Verizon's IPhone"</li>
<li>08:24 "Verizon big winner from having iPhone? Not so fast"</li>
<li>08:25 "McCourt Says Verizon's Network Can Handle Apple's IPhone: Video"</li>
<li>08:36 "How Important Is Offering the iPhone on Verizon to Apple?"</li>
<li>08:45 "[$$] ETF Play of the Day: XLK"</li>
<li>09:02 "[video] AM Report: Chinese Stealth Jet Takes Flight [8.0 min]"</li>
<li>09:06 "Three Things to Watch For at Verizon iPhoneapalooza"</li>
<li>09:29 "Verizon iPhone Faces Even Bigger Foe"</li>
<li>09:33 "Trujillo Says 'Many Questions' Remain For Verizon IPhone: Video"</li>
<li>09:45 "Verizon IPhone Event Live Blog"</li>
<li>09:45 "[video] News Hub: Verizon iPhone: How will AT&T Respond? [2.7 min]"</li>
<li>09:52 "[video] Verizon's iPhone Buzz"</li>
<li>09:54 "Mobile Broadband Subs Seen Hitting 1 Billion In 2011"</li>
<li>09:58 "[audio] Opening Bell from MarketWatch Radio Network [1.0 min]"</li>
<li>10:03 "Video: Is the Verizon iPhone a Threat to Android?"</li>
<li>10:05 "Verizon iPhone expected to have unlimited data plan"</li>
<li>10:14 "[video] iPhone or Not, Verizon Is A Winner"</li>
<li>10:19 "Verizon iPhone: Live Blog"</li>
<li>10:37 "Verizon: Stay Tuned"</li>
<li>10:43 "Mobile Broadband Subscriptions to Surpass 1 Billion in 2011"</li>
<li>10:47 "LIVE: The Verizon iPhone Event"</li>
<li>10:54 "LIVE FROM NEW YORK: Verizon Gets the iPhone"</li>
<li>10:54 "Verizon iPhone: Will It Fix Verizon's $100 Billion Conundrum?"</li>
<li>11:10 "Verizon to Partner With Apple, Start Selling IPhone Next Month"</li>
<li>11:14 "Verizon and Apple Make It Official"</li>
<li>11:15 "Verizon Wireless to Start IPhone Sales Next Month, Ending AT&T Exclusivity"</li>
<li>11:15 "iPhone Breaks AT&T Marriage Today for Verizon"</li>
<li>11:18 "Verizon to launch iPhone 4 in February"</li>
<li>11:20 "Verizon iPhone: The basics"</li>
<li>11:24 "The Biggest Surprise About the Verizon iPhone: It's a Mobile Hotspot"</li>
<li>11:24 "Verizon to Carry Apple iPhone, Ending Months of Anticipation"</li>
<li>11:30 "Verizon Will Be Selling The iPhone Starting February 10th"</li>
<li>11:38 "UPDATE 2-Verizon Wireless to sell iPhone in February"</li>
<li>11:39 "PHOTOS: Verizon iPhone Has Different (Better?) Antenna Design Than AT&T iPhone"</li>
<li>11:39 "Verizon, AT&T, And Apple: Guess Which One's Stock Isn't Tanking Today"</li>
<li>11:43 "Verizon Wireless to Offer iPhone 4 in February"</li>
<li>11:45 "Verizon iPhone: 'It Begins'"</li>
<li>11:50 "Verizon to begin selling iPhone 4 in February"</li>
<li>11:51 "Stocks Led by Energy, Conglomerates"</li>
<li>11:51 "Verizon to sell Apple iPhone"</li>
<li>11:51 "Verizon: Began Discussions With Apple In 2008; Tim Cook Talks"</li>
<li>11:54 "Verizon Wireless to sell iPhone in February"</li>
<li>12:08 "Verizon iPhone Will Not Work Overseas"</li>
<li>12:13 "UPDATE 3-Verizon Wireless to sell iPhone in February"</li>
<li>12:16 "FACTBOX - AT&T and Verizon wage battle over iPhone"</li>
<li>12:17 "AT&T Preps for Verizon iPhone With New Ads, Price Cuts"</li>
<li>12:19 "Apple's Phil Schiller on Building the Verizon iPhone"</li>
<li>12:20 "AT&T Shrugs Off Verizon iPhone"</li>
<li>12:20 "COMPARISON: Verizon Vs. AT&T iPhone"</li>
<li>12:26 "FACTBOX - AT&T and Verizon wage battle over iPhone"</li>
<li>12:28 "Verizon iPhone: Yep, The Real Target is Google."</li>
<li>12:30 "AT&T: Five Reasons the Sky Won't Fall When iPhone Goes to Verizon"</li>
<li>12:35 "[audio] Midday Update from MarketWatch Radio Network [1.0 min]"</li>
<li>12:37 "FACTBOX - AT&T and Verizon wage battle over iPhone"</li>
<li>12:37 "Verizon Wireless to sell iPhone in February"</li>
<li>12:43 "Verizon to start selling iPhone on Feb. 10"</li>
<li>12:48 "Corrected: Factbox: AT&T and Verizon wage battle over iPhone"</li>
<li>12:59 "Verizon Wireless to sell iPhone in February"</li>
<li>13:05 "Verizon iPhone set for Feb. 10 release"</li>
<li>13:05 "Verizon to start selling iPhone Feb. 3"</li>
<li>13:07 "Moore Says 3 Million AT&T IPhone Users May Go to Verizon: Video"</li>
<li>13:11 "Cook Says Apple-Verizon Cooperation 'Just the Beginning': Video"</li>
<li>13:12 "COMPARISON: Verizon Vs. AT&T iPhone"</li>
<li>13:18 "Most active New York Stock Exchange-traded stocks"</li>
<li>13:18 "Summary Box: Verizon to start selling iPhone"</li>
<li>13:18 "[video] Apple's iPhone 4 Comes to Verizon [1.4 min]"</li>
<li>13:22 "Apple/Verizon Pt. 2: Should I stay or should I go now?"</li>
<li>13:23 "FACTBOX: AT&T and Verizon wage battle over iPhone"</li>
<li>13:23 "Halftime: Verizon iPhone To Drag Down Verizon Stock 10%?"</li>
<li>13:29 "Wireless Struggles Against FCC Over Possible ""Bill Shock"" Rules"</li>
<li>13:34 "Verizon big winner from having iPhone? Not so fast"</li>
<li>13:43 "FACTBOX: AT&T and Verizon wage battle over iPhone"</li>
<li>13:43 "US stocks up on strong earnings [at Financial times]"</li>
<li>13:45 "Verizon to start selling iPhone on Feb. 10"</li>
<li>13:49 "Apple: No LTE iPhone This Year?"</li>
<li>13:53 "AT&T: We're ""Evaluating"" The New iPhone Mobile Hotspot Feature"</li>
<li>13:58 "Apple: Surprise! Street Expresses Delight With VZ iPhone"</li>
<li>14:02 "Did Apple disappoint with no 4G iPhone?"</li>
<li>14:04 "Verizon to Sell Apple IPad That Connects Directly to Its Network"</li>
<li>14:05 "Verizon begins selling iPhone on Feb. 3"</li>
<li>14:05 "Verizon iPhone sales begin Feb. 3"</li>
<li>14:09 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:10 "UPDATE 4-Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:13 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:25 "[audio] Before the Close from MarketWatch Radio Network [1.0 min]"</li>
<li>14:27 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:27 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>14:33 "Market Snapshot: U.S. stocks rise on earnings optimism"</li>
<li>14:33 "Verizon to start selling iPhone on Feb. 10"</li>
<li>14:41 "Analysts Go Out on Limb, Predict Verizon iPhone Will Be Big for Apple"</li>
<li>14:50 "Motorola: Opportunity In Lack Of LTE iPhone"</li>
<li>14:57 "An iPhone For Valentine's Day and iPad 2 Update"</li>
<li>14:58 Verizon</li>
<li>15:05 "Verizon iPhone sales begin Feb. 3"</li>
<li>15:05 "Verizon iPhone set for Feb. 10 release"</li>
<li>15:09 "How does the Verizon iPhone affect Android and AT&T?"</li>
<li>15:09 "More talk of smartphones diving into featurephone ground"</li>
<li>15:09 "Verizon iPhone unlimited data won't last forever"</li>
<li>15:09 "Why the Verizon 'dream phone' matters"</li>
<li>15:38 "5 'Dogs of the Dow' Worth Betting On"</li>
<li>15:38 "[video] iPhone: No wonders for Verizon stock"</li>
<li>15:50 "Apple: Jefferies Ups Tgt To $450; LTE iPhone In October?"</li>
<li>15:51 "Happy Verizon iPhone Day!"</li>
<li>15:58 "Verizon to start selling iPhone on Feb. 10"</li>
<li>16:06 "Verizon iPhone Is Not Mobile Messiah: Analyst"</li>
<li>16:10 "Stocks March Higher As Earnings Season Begins"</li>
<li>16:12 "Verizon's Daniel Mead Doesn't Expect IPhone Price War: Video"</li>
<li>16:17 "Verizon to start selling iPhone on Feb. 10"</li>
<li>16:18 "Android to Lose From Verizon iPhone?"</li>
<li>16:20 "Investing for Income in 2011 with Stocks and Options Part 3: High-Yield Stocks"</li>
<li>16:28 "Daley Has $7.7 Million of JPMorgan Stock to Divest on Way to White House"</li>
<li>16:30 "Apple: Reasons For An LTE iPhone This Year"</li>
<li>16:32 "Don't Buy The Verizon iPhone Yet, Unless You NEED A New Phone ASAP"</li>
<li>16:50 "Stronger earnings reports push stocks higher"</li>
<li>17:04 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>17:08 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>17:22 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>17:22 "Verizon Wireless ends long wait for iPhone fans"</li>
<li>17:41 "California Reclaims 48,000 State-Owned Mobile Phones"</li>
<li>17:43 "[video] News Hub: Verizon iPhone - What's The Actual Cost? [3.0 min]"</li>
<li>17:54 "Stronger earnings reports push stocks higher"</li>
<li>17:55 "Factbox: AT&T and Verizon wage battle over iPhone"</li>
<li>18:09 "All Our iPhone Trades, Right Here & Now"</li>
<li>18:09 "Most active New York Stock Exchange-traded stocks"</li>
<li>18:44 "Doherty Sees 'Exodus' of AT&T IPhone Users to Verizon: Video"</li>
<li>20:21 "MiFi Come, MiFi Go: Verizon Wireless To Sell iPads With 3G Connectivity"</li>
<li>22:02 "Tuesday ETF Roundup: UNG Surges on Weather, IYZ Tumbles on Increased Competition"</li>
<li>23:38 "Adding Up Parts at Goldman Sachs"</li>
</ul><br />
<h4>Merck (MRK): Thursday January 13th, 2011</h4><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7Ga4cs0UzOuiVWMaZh4Ig8k51yxWpbPFpQpt_SBTKdHkll8OjHGP-gpVikVDdCVERgokaDZ85opU06j9hmuO3UqEKBA3CQKGjQ6f7ZzSvsShdxYcjnK5gDjFMgWaDAN7A0CfM18hPfO4-/s1600/mrk-data.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="161" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7Ga4cs0UzOuiVWMaZh4Ig8k51yxWpbPFpQpt_SBTKdHkll8OjHGP-gpVikVDdCVERgokaDZ85opU06j9hmuO3UqEKBA3CQKGjQ6f7ZzSvsShdxYcjnK5gDjFMgWaDAN7A0CfM18hPfO4-/s400/mrk-data.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-ma5YGHt2lLh7ufyZFKW3ZCnP2Ug1TRG3jY8vPwmS7f5t1h-Qys4v0z33NbqlFzp2GTEOIX6oc4RZoaXka3Nweh3S9QJUxVWaWEHPSGkUY_slxW_qrhyphenhyphen6TPDtTKtPkCod4iX1FKzchXYc/s1600/mrk-deltas.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-ma5YGHt2lLh7ufyZFKW3ZCnP2Ug1TRG3jY8vPwmS7f5t1h-Qys4v0z33NbqlFzp2GTEOIX6oc4RZoaXka3Nweh3S9QJUxVWaWEHPSGkUY_slxW_qrhyphenhyphen6TPDtTKtPkCod4iX1FKzchXYc/s400/mrk-deltas.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2P8miv8Es3A3t5pHuJxMkqXSJLbSJw_km23bSQzyFUSHLUHdIaQKJs8rgwHXLRnVxiddxJa_eWyvMNcvd2EnJFrikQhlsG9NRAogzCH1nwQsmZXzPaVMfwEnibGkU2hZmkk8ehl0lm1cb/s1600/mrk-price.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2P8miv8Es3A3t5pHuJxMkqXSJLbSJw_km23bSQzyFUSHLUHdIaQKJs8rgwHXLRnVxiddxJa_eWyvMNcvd2EnJFrikQhlsG9NRAogzCH1nwQsmZXzPaVMfwEnibGkU2hZmkk8ehl0lm1cb/s400/mrk-price.png" width="400" /></a></div><br />
<ul><li>08:58 "JP Morgan Healthcare: Sanofi-Aventis, UCSF in Research Pact"</li>
<li>09:20 "Merck Statement on Changes to Clinical Studies of Vorapaxar"</li>
<li>09:41 "U.S. stocks open lower as joblessness rises"</li>
<li>09:41 "UPDATE 1-Merck stops giving clot drug in study, shares drop"</li>
<li>09:47 "Merck Drags on Stocks"</li>
<li>09:49 "Merck tumbles on drug study woes"</li>
<li>09:55 "Market Snapshot: U.S. stocks open lower as joblessness rises"</li>
<li>09:58 "Merck Blood-Thinner Study Ends in Stroke Patients, Limiting Possible Sales"</li>
<li>09:58 "Stocks Fall Slightly After Jobless Claims, PPI"</li>
<li>10:56 "Movers & Shakers: Thursday?s biggest gaining and declining stocks"</li>
<li>11:06 "Idaho in Drug Settlement With Warrick, Schering-Plough"</li>
<li>11:10 "Stocks Weighed by Healthcare Sector"</li>
<li>11:19 "[video] Markets Hub: US Stocks Fall As Merck Leads Decline [2.9 min]"</li>
<li>11:28 "FDA seeks less acetaminophen in prescription drugs"</li>
<li>11:34 "Commodities Look 'Very Attractive' Now: Kass"</li>
<li>11:41 "The Cheaper Way to Build the Best Portfolio"</li>
<li>11:43 "Biotech Stocks: Merck leads drug stocks lower"</li>
<li>11:43 "UPDATE 3-Merck clot drug seen unfit for stroke, shares fall"</li>
<li>11:47 "Merck Pipeline Setback: Big Pharma Losers"</li>
<li>11:49 "Merck clot drug seen unfit for stroke, shares fall"</li>
<li>11:53 "US STOCKS-Market flat, pares losses from jobless claims"</li>
<li>11:54 "Markets Going Nowhere After Weak US Data: Here Are The 10 Trades To Watch"</li>
<li>11:56 "Merck Learns There Is No Magic Pill"</li>
<li>11:56 "U.S. stocks bounce off lows; oil turns up"</li>
<li>12:16 "Bond Report: Treasurys rise after Fed purchases"</li>
<li>12:27 "Merck Shares Drop After Anti-Clotting Drug Trials Curtailed"</li>
<li>12:39 "Market Snapshot: U.S. stocks mixed as joblessness rises"</li>
<li>12:45 "Market flat, pares losses from jobless claims"</li>
<li>12:47 "A Painful Day for Merck"</li>
<li>13:05 "Merck tumbles on drug study update"</li>
<li>13:10 "US STOCKS-Market flat on weak jobless claims, Marathon's rise"</li>
<li>13:12 "Market flat, pares losses from jobless claims"</li>
<li>13:20 "Jobless claims weigh on US equities"</li>
<li>13:24 "Market flat on weak jobless claims, Marathon's rise"</li>
<li>13:29 "Dealpolitik: The Useful Corruption of Shareholder Lawsuits"</li>
<li>13:44 "Does Drug Setback Call Merck's Schering-Plough Purchase Into Question?"</li>
<li>13:47 "UPDATE 4-Merck clot drug seen unfit for stroke, shares fall"</li>
<li>13:48 "Market flat on weak jobless claims, Marathon's rise"</li>
<li>14:27 "Market Snapshot: Merck leads stocks lower"</li>
<li>14:35 "Merck Hurts, Stocks Stagger After Jobless Claims"</li>
<li>14:45 "Drug Study Torpedos Merck Stock"</li>
<li>14:52 "Bond Report: Treasurys lifted by Fed buys; 30-year sale in line"</li>
<li>14:55 "Jan. 13: Unusual Volume Leaders"</li>
<li>15:00 "Credit Suisse: All Isn't Lost for Merck"</li>
<li>15:12 "Merck weighs on Dow, market eyes earnings"</li>
<li>15:23 "This Trendy Pharma Is Making a Stylish Move"</li>
<li>15:36 "The Three Most Profitable Healthcare Trends of 2011"</li>
<li>15:47 "Market Snapshot: Stocks slide into close"</li>
<li>15:55 "Copying biotech medicine attracts more drugmakers"</li>
<li>16:00 "Stocks Slip Up On Awful US Data: Here's What You Need To Know"</li>
<li>16:08 "US STOCKS SNAPSHOT-Wall St dips on Merck, commodities"</li>
<li>16:09 "The Big Culprit In Today's DOW Drop"</li>
<li>16:09 "UPDATE 1-FDA seeks less acetaminophen in prescription drugs"</li>
<li>16:10 "U.S. stocks end lower as joblessness rises"</li>
<li>16:27 "US STOCKS-Merck, materials companies drag Wall St lower"</li>
<li>16:27 "[audio] Closing Bell from MarketWatch Radio Network [1.0 min]"</li>
<li>16:55 "Bond Report: Treasurys rise with jobless claims, auctions? end"</li>
<li>16:56 "[video] PM Report: Economists Turn Upbeat on Recovery [10.6 min]"</li>
<li>16:59 "[$$] Stocks in the Spotlight Thursday"</li>
<li>17:06 "US STOCKS-Merck, materials drag Wall St lower, Intel up late"</li>
<li>17:09 "[video] News Hub: Dow Dragged Down by Merck [1.3 min]"</li>
<li>17:11 "Market Snapshot: Stocks end lower on weak materials, drug shares"</li>
<li>18:29 "The Fed Takes Credit: Dave's Daily"</li>
<li>18:34 "[audio] After the Bell from MarketWatch Radio Network [1.0 min]"</li>
<li>18:39 "Trouble With Thrombin Inhibitor Takes a Toll on Merck Shares"</li>
<li>18:59 "Thursday ETF Roundup: UNG Sinks on Natural Gas Data, BLV Rises on Bond Boost"</li>
<li>20:06 "[$$] Options Traders Target Merck After Drug Study Halted"</li>
<li>20:26 "HK stocks seen lower on weak materials sector"</li>
</ul><br />
<h4>Bank of America (BAC): Friday January 14th, 2011</h4><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhu4R_ixFP41d84hn3tH051en5TpPQapUiVQo8UeBvf948hsl8VfaBB_tsb3JtFAkaVK02K6uSZMXi-KFKuiUuClXm-L36ChC1MuGaunPn67ZWcr_0745A5MhQvdpPbiYmkx4k8RzFjbQzs/s1600/bac-data.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhu4R_ixFP41d84hn3tH051en5TpPQapUiVQo8UeBvf948hsl8VfaBB_tsb3JtFAkaVK02K6uSZMXi-KFKuiUuClXm-L36ChC1MuGaunPn67ZWcr_0745A5MhQvdpPbiYmkx4k8RzFjbQzs/s400/bac-data.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLkiY4yNrJc3t9wJ9BE3lVzzEH2dWGxdReNUv5t2wB5XFvHVUJ3BtK02QVVN_qd5ZeBn7nXZnNznNoykvGtL7lCNtO7NuL5xnEcIk4S-wgcI6xr9WXoXnJHNfIZyywVhDka8R1Q7lvwbdj/s1600/bac-deltas.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLkiY4yNrJc3t9wJ9BE3lVzzEH2dWGxdReNUv5t2wB5XFvHVUJ3BtK02QVVN_qd5ZeBn7nXZnNznNoykvGtL7lCNtO7NuL5xnEcIk4S-wgcI6xr9WXoXnJHNfIZyywVhDka8R1Q7lvwbdj/s400/bac-deltas.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0elcPLjEgLBkCYoiX4ousAfjzfadFSSoSVdHFvvLUVAm9yu3SmiuZgtn1gy7blxXS_v4zoJpSlEcCj3tPVbLqHs7Gc7xn3MVzi2NLxPMXaWKb9hoVtNIgHGD5hWzaiH-q5l1hYntUtZ_0/s1600/bac-price.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0elcPLjEgLBkCYoiX4ousAfjzfadFSSoSVdHFvvLUVAm9yu3SmiuZgtn1gy7blxXS_v4zoJpSlEcCj3tPVbLqHs7Gc7xn3MVzi2NLxPMXaWKb9hoVtNIgHGD5hWzaiH-q5l1hYntUtZ_0/s400/bac-price.png" width="400" /></a></div><br />
<ul><li>00:01 "Google, Renault, RBS, Stanford, Kissel, Madoff in Court News"</li>
<li>00:01 "Lehman, Madoff, FGIC, Chemtura, FairPoint, Quigley: Bankruptcy"</li>
<li>00:02 "Geithner starts talks on tax overhaul"</li>
<li>01:07 "Morgan Stanley's Gorman shakes up management"</li>
<li>01:25 "Morgan Stanley's Gorman shakes up management"</li>
<li>02:06 "Banks Are Poised to Pay Dividends After 3-Year Gap"</li>
<li>03:12 "UPDATE 1-India govt to decide on IOC share sale in 3-4 mths"</li>
<li>04:01 "What's Driving Earnings Higher?"</li>
<li>06:00 "5 Bank Stocks Under $5"</li>
<li>06:00 "5 Land Mines in Fourth Quarter Bank Earnings"</li>
<li>06:00 "[video] Hold, Don't Buy Bank Stocks: Fund Manager"</li>
<li>06:59 "Kissel Was Argumentative, Easily Upset, Sister-In-Law Testifies"</li>
<li>07:00 "Filling In FINRA's Gaps"</li>
<li>07:00 "General Electric Earnings: What to Expect"</li>
<li>07:00 "Worst-Performing Financial CEO of 2010"</li>
<li>07:16 "[$$] Earnings Scorecard: Commercial and Investment Banks"</li>
<li>07:18 "JPMorgan profit surges despite mortgage hit"</li>
<li>07:36 "JPMorgan's Fourth Quarter Profit Rises 47% to $4.8 Billion"</li>
<li>07:48 "RPT-UPDATE 4-Morgan Stanley's Gorman shakes up management"</li>
<li>07:59 "J.P. Morgan's Fourth-Quarter Profit Jumps 47%"</li>
<li>08:04 "Floods take major toll on Australian farms, mines"</li>
<li>08:08 "A Stock-Bond Hybrid That Flopped"</li>
<li>08:08 "Big Banks with Big Legal Bills"</li>
<li>08:08 "Home Depot's Fix-It Lady"</li>
<li>08:08 "Why the Private Sector Still Isn't Hiring"</li>
<li>08:12 "Citigroup, Bank of America Active Premarket After JPM Report"</li>
<li>08:19 "What's On: JPMorgan, Intel and Corporate Taxes"</li>
<li>08:24 "UPDATE 4-JPMorgan profit beats, helped by reserve release"</li>
<li>08:26 "Hybrid Debt, Materials Curbs, Tax Cheats: Compliance"</li>
<li>08:26 "JPMorgan profit beats, helped by reserve release"</li>
<li>08:37 "Deals of the Day: Groupon Talks IPO"</li>
<li>08:53 "GLOBAL MARKETS WEEKAHEAD-Reality checks ahead for investors"</li>
<li>08:54 "J.P. Morgan Earnings: Were They As Good as They Looked?"</li>
<li>08:55 "J.P. Morgan Earnings: Still Building Litigation Warchest"</li>
<li>09:01 "[$$] ETF Play of the Day: KBE"</li>
<li>09:10 "Foreclosure Wave Cresting, May Never Break"</li>
<li>09:40 "Earnings Jump At JPMorgan, But Street Stifled By China"</li>
<li>09:42 "U.S. Probes Banks' Mortgage Practices"</li>
<li>09:46 "JPMorgan profit rises 47 percent"</li>
<li>09:49 "J.P. Morgan: Mortgage Fights a 'Long, Ugly Mess'"</li>
<li>09:52 "ManTech International - Upgrades & Downgrades"</li>
<li>09:56 "Lbma Says Merrill Lynch Reclassified as Options Market Maker"</li>
<li>09:57 "UBS Upgrades Financials"</li>
<li>10:04 "American Capital prices share offer at $28 each"</li>
<li>10:09 "Schwab Takes High-Profile Stand Opposing Self-Regulatory Organization"</li>
<li>10:12 "BofA Suffers Online Glitches"</li>
<li>10:16 "JPMorgan: Strong Earnings, Muted Response"</li>
<li>10:23 "Stocks edge up after tepid inflation report"</li>
<li>10:33 "JPM Dividend: Jamie Dimon Balks"</li>
<li>10:34 "Your Index Fund Is Making a Bad Investment"</li>
<li>10:43 "Apple's Magic Share Price: $422"</li>
<li>10:45 "JPMorgan Chase Tops 4Q Estimates. Citigroup Is Up Next."</li>
<li>10:58 "Citigroup: Uncle Sam to Get Out, Completely"</li>
<li>11:01 "[$$] JPMorgan Chase Delivers Insight"</li>
<li>11:05 "Affiliated Managers gets $750M credit line"</li>
<li>11:08 "A Wave of Mergers And Acquisitions for Northeast And Mid-Atlantic Banks: An Exclusive Interview With Stephen M. Moss Of Janney Montgomery Scott LLC"</li>
<li>11:11 "UPDATE 6-JPMorgan profit rises 47 pct, beating estimates"</li>
<li>11:11 "[$$] Big Banks May Boost Dividends in 2011"</li>
<li>11:13 "New Zealand bourse winds up AXE trading venue"</li>
<li>11:20 "Greenspan, Fed Saw Threat of Stagflation After Katrina Devastation in 2005"</li>
<li>11:25 "A Wall Street Mystery: Where Was Goldman's CEO?"</li>
<li>11:29 "Goldman Lost an Extra $5B in Prop Trading: Report"</li>
<li>11:29 "JPMorgan profit rises 47 percent"</li>
<li>11:49 "[video] How JPMorgan's Quarter Affects Bank Stocks"</li>
<li>11:53 "Banks lead stocks higher, led by JPMorgan Chase"</li>
<li>12:03 "JPMorgan Earnings Lift Bank Stocks"</li>
<li>12:03 "John Paulson's Top Picks"</li>
<li>12:04 "CPI Shows Input-Cost Inflation Tempered By Overcapacity"</li>
<li>12:05 "Bank of America website having problems"</li>
<li>12:05 "JPMorgan Chase profit up 47%"</li>
<li>12:08 "Earnings schedule for week of 1/17/2011"</li>
<li>12:10 "CNNMoney stock market report -- Jan. 14, 2011"</li>
<li>12:26 "Banks lead stocks higher, led by JPMorgan Chase"</li>
<li>12:29 "J.P. Morgan?s profit jumps 47%"</li>
<li>12:36 "[$$] ETFs for the Dividend-Minded Investor"</li>
<li>13:44 "UPDATE 7-JPMorgan beats, sets bullish tone for bank earnings"</li>
<li>13:48 "AIG recap deal closes, focus moves to share sale"</li>
<li>13:48 "JPMorgan sets bullish tone for bank earnings"</li>
<li>13:50 "[video] J.P. Morgan Earnings Rise 47%"</li>
<li>13:52 "New Movie Revisits the Last 24 Hours of Lehman Brothers, or a Very Similar Firm"</li>
<li>13:52 "[$$] J.P. Morgan Gives Dow Modest Boost"</li>
<li>13:54 "Fed?s Tarullo Tells CNBC No Reason to Adjust Asset Purchases"</li>
<li>13:55 "Should You Buy 2010's Best Investments?"</li>
<li>14:00 "Banking On A Better 2011"</li>
<li>14:02 "[video] Rush Card"</li>
<li>14:05 "JPMorgan Credit Swaps Climb as Bank Results Rest on Reserves"</li>
<li>14:06 "How Celebrities Buy And Sell Homes"</li>
<li>14:07 "A female quota at Davos? Really?"</li>
<li>14:26 "JPMorgan Rises On Double Beat"</li>
<li>14:40 "Bank of America, Wells Fargo May Lose Under Plan to Simplify Mortgages"</li>
<li>14:43 "PNC, Capital One Advance After JPMorgan Posts Record Profit"</li>
<li>14:45 "Bank of America online banking down for some users"</li>
<li>14:45 "Banks take stock indexes higher"</li>
<li>14:56 "Financial Stocks: Financial shares higher after J.P. Morgan results"</li>
<li>15:01 "Halftime: Trading Financials Ahead of Earnings Next Week"</li>
<li>15:05 "Bank of America website having problems"</li>
<li>15:10 "Bank of America website suffers outage"</li>
<li>15:16 "AIG recap deal closes, focus moves to share sale"</li>
<li>15:21 "JPMorgan beats, sets bullish tone for bank earnings"</li>
<li>15:27 "PREVIEW-UPDATE 1-Volcker rule tests new US systemic risk council"</li>
<li>15:29 "Bank of America online banking down for some users"</li>
<li>15:37 "Bank of America Downplays Web Problems"</li>
<li>15:55 "Volcker rule tests new US systemic risk council"</li>
<li>15:57 "AIG recap deal closes, focus moves to share sale"</li>
<li>15:57 "Bank of America Website is Down For Some"</li>
<li>15:59 "[$$] Treasurys Fall, Giving Up Earlier Rally; Little Change for Week"</li>
<li>16:10 "Assured Guaranty was only muni bond insurer in 2010"</li>
<li>16:17 "U.S. stocks post seventh week of gains"</li>
<li>16:18 "Great Depression Part Deux Averted But Bubble Machine Still Blows"</li>
<li>16:18 "Volcker rule tests new US systemic risk council"</li>
<li>16:23 "Tax talks get good start, companies say"</li>
<li>16:29 "Banks take stock indexes higher, led by JPMorgan"</li>
<li>16:31 "Dow, S&P 500 Extend Seven-Week Winning Streak"</li>
<li>16:31 "[$$] Interesting Rotation in the Big Banks"</li>
<li>16:36 "Banks lead S&P 500 to seventh week of gains"</li>
<li>16:52 "Stocks Reach New Records"</li>
<li>16:54 "Market Snapshot: U.S. stocks rise to 30-month highs"</li>
<li>16:55 "Trading Radar: Key Earnings, China Headline the Week"</li>
<li>17:41 "JPMorgan beats, sets bullish tone for bank earnings"</li>
<li>18:15 "AIG recapitalization deal closes, share sale looms"</li>
<li>18:32 "Wall St Week Ahead: Investors crave more strong bank results"</li>
<li>18:46 "GLOBAL MARKETS WEEKAHEAD - Reality checks ahead for investors"</li>
<li>18:46 "US STOCKS - Banks lead S&P 500 to seventh week of gains"</li>
<li>18:46 "Volcker rule tests new US systemic risk council"</li>
<li>20:55 "AIG recapitalization deal closes, share sale looms"</li>
<li>20:55 "JPMorgan beats, sets bullish tone for bank earnings"</li>
</ul><br />
<h4>JP Morgan (JPM): Friday January 14th, 2011</h4><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgV2m-3_N6Nk0ZlS4J1ieNSL4PNpEq41rssWfiDBegeR0hFKSUWiNgiu-FjkZ2GbHZhWtV7gu1GnBFDyECoorxS2cpQgRgY6jAxsJC0EVvVZY0P_tqZvev91pHxaVcvekdZg1tAN4BU_0Lq/s1600/jpm-data.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgV2m-3_N6Nk0ZlS4J1ieNSL4PNpEq41rssWfiDBegeR0hFKSUWiNgiu-FjkZ2GbHZhWtV7gu1GnBFDyECoorxS2cpQgRgY6jAxsJC0EVvVZY0P_tqZvev91pHxaVcvekdZg1tAN4BU_0Lq/s400/jpm-data.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNXnmRsQm2ojkRCKHZX7S5N-nfbfDAvXyTK0xORF0hBKukjwXa64UtVVtbNDY6eYxi5Me8eDF9Sk-f3iUPC_ZaqFRZrU0yLhKcsjwgrawsjX4BkfLDSnC383mtwSQS2c4CoYEaBmfWfBpa/s1600/jpm-deltas.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNXnmRsQm2ojkRCKHZX7S5N-nfbfDAvXyTK0xORF0hBKukjwXa64UtVVtbNDY6eYxi5Me8eDF9Sk-f3iUPC_ZaqFRZrU0yLhKcsjwgrawsjX4BkfLDSnC383mtwSQS2c4CoYEaBmfWfBpa/s400/jpm-deltas.png" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1IEDa79CzIJlqof8qMD9UVOeCY3QqqzrZhk-JwspQU_vMz-R9lPPd13c3F_sJLjqze3mAJO4_HKBl0lkiCCrXSY91XkwcACGCS70Pbi-sQn1oDcCEEipYrU1AjiY8lDe3DnTMKhfRDKcq/s1600/jpm-price.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1IEDa79CzIJlqof8qMD9UVOeCY3QqqzrZhk-JwspQU_vMz-R9lPPd13c3F_sJLjqze3mAJO4_HKBl0lkiCCrXSY91XkwcACGCS70Pbi-sQn1oDcCEEipYrU1AjiY8lDe3DnTMKhfRDKcq/s400/jpm-price.png" width="400" /></a></div><br />
<ul><li>07:29 "JPMorgan Profit Beats on Narrowing Loan Losses"</li>
<li>07:29 "JPMorgan reports jump in quarterly profit"</li>
<li>07:33 "US STOCKS-Futures fall, JPMorgan profit jumps"</li>
<li>07:34 "JPMorgan Net Rises 47% on Lower Credit Costs, Tops Estimates"</li>
<li>07:35 "Futures Lower Despite Earnings Beats"</li>
<li>07:35 "[audio] Before the bell from MarketWatch Radio Network [1.0 min]"</li>
<li>07:36 "JPMorgan's Fourth Quarter Profit Rises 47% to $4.8 Billion"</li>
<li>07:37 "J.P. Morgan posts quarterly profit of $4.8 billion"</li>
<li>07:39 "UPDATE 3-JPMorgan profit beats, helped by reserve release"</li>
<li>07:40 "The Good Leads"</li>
<li>07:41 "JPMorgan Chase's profit jumps 47 percent in 4Q"</li>
<li>07:41 "JPMorgan quarterly profits surge 47%"</li>
<li>07:42 "JPMorgan profit beats expectations"</li>
<li>07:42 "UPDATE 1-TABLE-JPMorgan fourth-quarter results"</li>
<li>07:45 "CNNMoney premarket report -- Jan. 14, 2011"</li>
<li>07:48 "RPT-UPDATE 4-Morgan Stanley's Gorman shakes up management"</li>
<li>07:56 "[$$] J.P. Morgan Profit Jumps 47%"</li>
<li>07:57 "Indications: U.S. stock futures point to soft open"</li>
<li>07:57 "Stock futures off, Dow, S&P seek 7th week of gains"</li>
<li>07:59 "J.P. Morgan's Fourth-Quarter Profit Jumps 47%"</li>
<li>08:26 "JPMorgan profit beats, helped by reserve release"</li>
<li>08:28 "JPMorgan Beats Street, Net Rises 47%"</li>
<li>08:28 "Morning Take-Out"</li>
<li>08:30 "Minyanville's T3 Morning Call: JPM, INTC Earnings Beats Can't Lift Futures"</li>
<li>08:30 "[$$] The Day Ahead: Follow the Leaders"</li>
<li>08:31 "JP Morgan results ahead of forecasts"</li>
<li>08:36 "[$$] It's All Baked In"</li>
<li>08:41 "Deals wrap: JPMorgan surprises"</li>
<li>08:41 "[$$] Futures Stay Weak After U.S. Data"</li>
<li>08:43 "M&T Beats Estimates"</li>
<li>08:44 "US STOCKS-Futures briefly tick lower after CPI, retail sales"</li>
<li>08:45 "Futures Hold Losses After CPI, Retail Sales"</li>
<li>08:46 "[$$] Will Intel Be the Downside Catalyst?"</li>
<li>08:52 "Live coverage of JPMorgan?s earnings call"</li>
<li>08:54 "J.P. Morgan Earnings: Were They As Good as They Looked?"</li>
<li>08:55 "J.P. Morgan Earnings: Still Building Litigation Warchest"</li>
<li>08:56 "JPMorgan Chase's profit jumps 47 percent in 4Q"</li>
<li>08:57 "JPMorgan reports jump in quarterly profit"</li>
<li>09:38 "CNNMoney stock market report -- Jan. 14, 2011"</li>
<li>09:38 "JPMorgan's Biggest Cost? Pay, Not Mortgages"</li>
<li>09:38 "UPDATE 5-JPMorgan profit rises 47 pct, beating estimates"</li>
<li>09:40 "Earnings Jump At JPMorgan, But Street Stifled By China"</li>
<li>09:41 "Financials open higher after J.P. Morgan earnings"</li>
<li>09:41 "Stocks Slip at Open After Mixed Economic News"</li>
<li>09:42 "U.S. Probes Banks' Mortgage Practices"</li>
<li>09:45 "Unique Convert Offering From Dendreon via JPMorgan"</li>
<li>09:46 "JPMorgan profit rises 47 percent"</li>
<li>09:47 "Friday Look Ahead: Intel, JPMorgan Earnings and Consumer Mood"</li>
<li>09:48 "J.P. Morgan?s profit jumps 47%"</li>
<li>09:49 "J.P. Morgan: Mortgage Fights a 'Long, Ugly Mess'"</li>
<li>09:50 "Stock Market Story: Jan. 14"</li>
<li>09:57 "JPMorgan Reports Q4 Results"</li>
<li>09:57 "Markets dip after JPMorgan earns, retail sales"</li>
<li>09:57 "UBS Upgrades Financials"</li>
<li>10:16 "JPMorgan: Strong Earnings, Muted Response"</li>
<li>10:23 "Stocks edge up after tepid inflation report"</li>
<li>10:24 "[$$] JPMorgan Worries Down the Drain"</li>
<li>10:25 "[$$] Mixed Start"</li>
<li>10:28 "US open: Dow trades flat"</li>
<li>10:30 "[$$] A Short-Term Retail Play"</li>
<li>10:31 "China's inflation move weighs on Wall St"</li>
<li>10:33 "JPM Dividend: Jamie Dimon Balks"</li>
<li>10:45 "[$$] Dip-Buyers' Delight"</li>
<li>10:47 "UPDATE 1-JPMorgan Q4 commods risk steady after market rally"</li>
<li>10:54 "Friday's ETF to Watch: Merrill Lynch Regional Bank HOLDR"</li>
<li>10:55 "J.P. Morgan banker pay slips -- to $369,651"</li>
<li>10:57 "Markets flat as JPMorgan results offset Intel"</li>
<li>10:58 "Citigroup: Uncle Sam to Get Out, Completely"</li>
<li>10:58 "Financial Stocks: Financial shares higher after J.P. Morgan results"</li>
<li>11:08 "A Wave of Mergers And Acquisitions for Northeast And Mid-Atlantic Banks: An Exclusive Interview With Stephen M. Moss Of Janney Montgomery Scott LLC"</li>
<li>11:11 "[$$] Big Banks May Boost Dividends in 2011"</li>
<li>11:16 "[$$] J.P. Morgan Bankers Earn Less"</li>
<li>11:20 "Greenspan, Fed Saw Threat of Stagflation After Katrina Devastation in 2005"</li>
<li>11:20 "JPMorgan Paces Bank Gains After Earnings"</li>
<li>11:25 "A Wall Street Mystery: Where Was Goldman's CEO?"</li>
<li>11:26 "Debit fee caps may hurt poorest customers: Dimon"</li>
<li>11:28 "Market Snapshot: Stocks edge up as J.P. Morgan lifts banks"</li>
<li>11:29 "Goldman Lost an Extra $5B in Prop Trading: Report"</li>
<li>11:29 "JPMorgan profit rises 47 percent"</li>
<li>11:29 "News Corp. in Talks to Buy Shine"</li>
<li>11:33 "JPM Earnings"</li>
<li>11:41 "Complacency, Fleeing Insiders Point to Earnings Season Drop"</li>
<li>11:42 "Gold Prices Tank on Moves to Curb Inflation"</li>
<li>11:42 "JPMorgan: Bonus Numbers Look Good But Bankers Are Wary"</li>
<li>11:46 "UPDATE 2-TABLE-JPMorgan fourth-quarter results"</li>
<li>11:49 "[video] How JPMorgan's Quarter Affects Bank Stocks"</li>
<li>11:53 "Banks lead stocks higher, led by JPMorgan Chase"</li>
<li>12:03 "JPMorgan Earnings Lift Bank Stocks"</li>
<li>12:05 "Bank of America website having problems"</li>
<li>12:05 "JPMorgan Chase profit soars 47%"</li>
<li>12:05 "JPMorgan Chase profit up 47%"</li>
<li>12:09 "A Buying Opportunity Before Stress-Test Round 2?"</li>
<li>12:10 "S&P 500 Is Poised to Drop; RBC, JPMorgan Differ on the Timing"</li>
<li>12:11 "[$$] Comeback After Dazzling Comeback"</li>
<li>12:12 "US STOCKS-Market rises as JPMorgan profit outweighs data"</li>
<li>12:26 "Banks lead stocks higher, led by JPMorgan Chase"</li>
<li>12:32 "[audio] Midday Update from MarketWatch Radio Network [1.0 min]"</li>
<li>12:39 "Hey, Jamie Dimon, You Sure About that Acronym?"</li>
<li>12:55 "Stocks hurt by global inflation concerns"</li>
<li>12:56 "China, U.S. Weigh on Europe's Markets"</li>
<li>13:10 "Market Snapshot: Stocks find footing J.P. Morgan lifts banks"</li>
<li>13:11 "US STOCKS-Wall St rises as JPMorgan earnings spur optimism"</li>
<li>13:14 "Spain, Portugal And A Virtuous Circle Of Bailouts"</li>
<li>13:20 "Most active New York Stock Exchange-traded stocks"</li>
<li>13:34 "4 Top Financials for 1Q from Bernstein Research"</li>
<li>13:36 "Bank Of America Website Down, 6 Hours And Counting"</li>
<li>13:43 "UPDATE 3-AIG recap deal closes, focus moves to share sale"</li>
<li>13:44 "JPMorgan Chase's income jumps 47 percent"</li>
<li>13:48 "AIG recap deal closes, focus moves to share sale"</li>
<li>13:48 "JPMorgan sets bullish tone for bank earnings"</li>
<li>13:50 "[video] J.P. Morgan Earnings Rise 47%"</li>
<li>13:52 "New Movie Revisits the Last 24 Hours of Lehman Brothers, or a Very Similar Firm"</li>
<li>13:53 "JPMorgan: JV Approval in China Positions Stock for Upside"</li>
<li>13:54 "Fed?s Tarullo Tells CNBC No Reason to Adjust Asset Purchases"</li>
<li>13:59 "Elisabeth Murdoch's Shine hires bank for review"</li>
<li>14:00 "Banking On A Better 2011"</li>
<li>14:03 "J P MORGAN CHASE & CO Files SEC form 8-K, Regulation FD Disclosure"</li>
<li>14:05 "JPMorgan Credit Swaps Climb as Bank Results Rest on Reserves"</li>
<li>14:06 "How Celebrities Buy And Sell Homes"</li>
<li>14:07 "A female quota at Davos? Really?"</li>
<li>14:10 "M&T Bank Tops Zacks Estimate"</li>
<li>14:17 "Summary Box: JPMorgan Chase's profit jumps in 4Q"</li>
<li>14:17 "What's Behind JP Morgan's $1.12 Shares"</li>
<li>14:20 "Hedge funds appeal ruling on WaMu trust securities"</li>
<li>14:26 "JPMorgan Rises On Double Beat"</li>
<li>14:31 "UK unmatched for bank bonus-bashing"</li>
<li>14:34 "VIX Falls to Three-Week Low After JPMorgan Posts Record Profit"</li>
<li>14:35 "US STOCKS-JPMorgan lifts Wall St, but banks due for pause"</li>
<li>14:40 "Bank of America, Wells Fargo May Lose Under Plan to Simplify Mortgages"</li>
<li>14:43 "PNC, Capital One Advance After JPMorgan Posts Record Profit"</li>
<li>14:45 "Banks take stock indexes higher"</li>
<li>14:50 "Explaining Confirmation"</li>
<li>14:58 "JPMorgan Chase Plans $3.25 Billion Debt Offering"</li>
<li>15:01 "Gold Prices Sink on Rising Inflation Concerns"</li>
<li>15:05 "Bank of America website having problems"</li>
<li>15:16 "AIG recap deal closes, focus moves to share sale"</li>
<li>15:17 "Bond Report: Treasurys fall as relief rally dissipates"</li>
<li>15:21 "JPMorgan beats, sets bullish tone for bank earnings"</li>
<li>15:22 "Market Snapshot: U.S. stocks rise as J.P. Morgan lifts banks"</li>
<li>15:27 "PREVIEW-UPDATE 1-Volcker rule tests new US systemic risk council"</li>
<li>15:39 "[$$] Crude Settles Above $91"</li>
<li>15:40 "Why Citi Won't Repeat JPMorgan's Success"</li>
<li>15:43 "What to Do With JP Morgan Now?"</li>
<li>15:50 "JPMorgan Chase Declares Preferred Stock Dividend"</li>
<li>16:15 "PNC, Capital One: Winners & Losers"</li>
<li>16:17 "U.S. stocks post seventh week of gains"</li>
<li>16:18 "Great Depression Part Deux Averted But Bubble Machine Still Blows"</li>
<li>16:18 "Volcker rule tests new US systemic risk council"</li>
<li>16:25 "[audio] Closing Bell from MarketWatch Radio Network [1.0 min]"</li>
<li>16:29 "Banks take stock indexes higher, led by JPMorgan"</li>
<li>16:30 "MarketWatch?s top 10 stories: Jan. 10-14"</li>
<li>16:31 "HCP, Kimco: REIT Broker Action"</li>
<li>16:31 "New Issue-JP Morgan sells $3.25 bln in 2 parts"</li>
<li>16:31 "US STOCKS-Banks lead S&P 500 to seventh week of gains"</li>
<li>16:31 "[$$] Interesting Rotation in the Big Banks"</li>
<li>16:32 "Coinstar, JPMorgan, Borders are big market movers"</li>
<li>16:48 "Stocks Buck Mixed Data to Head Higher"</li>
<li>16:50 "Buzz on the Street: Tech Stocks Continue to Lead the Way"</li>
<li>16:52 "Stocks Reach New Records"</li>
<li>16:54 "Market Snapshot: U.S. stocks rise to 30-month highs"</li>
<li>16:55 "[video] News Hub: Dow Ends at 2-1/2 Year High [1.5 min]"</li>
</ul>Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com0tag:blogger.com,1999:blog-4261954431421179908.post-49112051946399986152010-12-13T19:00:00.008+00:002012-09-23T12:53:43.417+01:00Ceci n'est ce pas une Haskell Monad TutorialWith apologies to the great surrealist painter <a href="http://en.wikipedia.org/wiki/Ren%C3%A9_Magritte">Magritte</a>, this really isn't a Haskell Monad Tutorial (HMT), or is it?<br />
<br />
If you are reading this, you are probably aware that the blogosphere is littered with HMTs. The progression usually goes like this:<br />
<br />
A programmer...<br />
<ol>
<li>undertakes to learn Haskell, </li>
<li>struggles for some period of time with understanding Monads, </li>
<li>suddenly has that "ah-ha" moment, and </li>
<li>writes a tutorial (convinced that they can finally explain it to the beginner better than anyone else who has gone before could).</li>
</ol>
In my case, I choose to diverge from this pattern (slightly):<br />
<br />
I...<br />
<ol>
<li>took it upon myself to learn Haskell</li>
<li>struggled with Monads for some time</li>
<li>kind of slowly and grudgingly eventually "got" monads, but not with any kind of dramatic "eureka!", more like a "oh... right... ok" kind of feeling (which was remarkably anti-climactic considering how much hair I'd pulled out over this), and </li>
<li>am writing not so much a tutorial as a: "Here's what I did to understand it. Hope it helps." </li>
</ol>
I am not going to talk about Monad laws, IO, or Maybe. I am not even going to try to concoct a flowery definition of what it is (like "contanier" or "computational context"). I am just going to give two examples, which I wrote, specificially with a view to clarifying my own understanding (as per the suggestion of Brian Beckman, see video link below).<br />
<br />
If this helps you, great. I wrote this to help me, and it served that purpose. It's called "learn by doing" (practical, rather than theoretical). I therefore must also suggest that you try this yourself, rather than just expect to get it by reading this. I'm willing to concede that there are probably serious problems with these definitions, given that I am not a schooled Haskell programmer, and don't know a lot about the conventions that serious people follow.<br />
<br />
I'd like to start by throwing down a few references which were helpful to me in my understanding:<br />
<ul>
<li><a href="http://channel9.msdn.com/Shows/Going+Deep/Brian-Beckman-Dont-fear-the-Monads">Brian Beckman: Don't Fear the Monads</a> (video)</li>
<li><a href="http://channel9.msdn.com/Blogs/Charles/C9-Lectures-Greg-Meredith-Monadic-Design-Patterns-for-the-Web-Introduction-to-Monads">Greg Meredith: Intro to Monads</a> (video)</li>
<li><a href="http://learnyouahaskell.com/a-fistful-of-monads">Learn You a Haskell for Great Good</a> (best "book" for beginners)</li>
<li><a href="http://james-iry.blogspot.com/search/label/monads">James Iry's blog: One Div Zero</a></li>
</ul>
Now, those folks really know what they're talking about. Look to them for a real education. But, if you still want to see my rudimentary examples, here they are...<br />
<br />
The two examples I implemented: the identity monad (the simplest of all!), and the state monad (because I really wanted to understand this for a project that I'm working on). Implementing the identity monad really helped me, because it's just so simple, yet has all the necessary properties of a "real" monad. State was far more difficult, mainly because of the fact that the monad wraps a function, rather than a value (although we know that these are the same thing in Haskell!), but once I cracked this, it really did "all make sense."<br />
<br />
The Identity Monad<br />
<br />
First, we need a type and type constructor (by Haskell convention I will use the same expression for both, which for this example is "Id").<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> newtype Id a = Id a deriving Show
</code></pre>
<br />
Note: the "deriving Show" bit is there just in case I want to print these to the console. It could have just been:<br />
<br />
Then I'm going to need a function to extract the value from the "Id" thingy:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> runId :: Id a -> a
runId (Id x) = x
</code></pre>
<br />
Simple enough, so far?<br />
<br />
Note: this could also have been acheived using Haskell's "record syntax" as follows (saving the trouble of writing the extra function):<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> newtype Id a = Id { runId :: a } deriving Show
</code></pre>
<br />
Now, the Monad bit. Don't blink, or you might miss it!<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> instance Monad Id where
return a = Id a
m >>= f = f (runId m)
</code></pre>
<br />
Identity monad... done!<br />
<br />
I'm not going to explain this in any great detail. You've probably read about "return" and "bind (>>=)" elsewhere. In bind: "m" is the monadic value, "f" is a function (a -> mb) and the bit on the right of the equals just says "run f on the value inside of m."<br />
<br />
Some functions that operate on Id thingies:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> idPlusBar :: Id String -&gt; Id String
idPlusBar x = do
y <- x
z <- Id (y ++ "bar")
return z
</code></pre>
<br />
This function operates on Id Strings appends "bar" to the value.<br />
<br />
Note: the function above could be written more simply as:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> idPlusBar x = Id $ (runId x) ++ "bar"
</code></pre>
<br />
but I included it in long form for illustration purposes.<br />
<br />
Here are a couple that operate on Id Ints:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> idPlusTwo :: Id Int -> Id Int
idPlusTwo x = Id $ (runId x) + 2
idTimesThree :: Id Int -> Id Int
idTimesThree x = Id $ (runId x) * 3
idComposed :: Id Int -> Id Int
idComposed = idPlusTwo . idTimesThree
</code></pre>
<br />
Now, here's a function ("main" in this case, as I can run this from the command line), which uses HUnit (you'll need to "import Test.HUnit") to test some assumptions:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> main = do
putStrLn "running tests"
-- Id String tests
let x = Id "foo"
let test1 = TestCase $ assertEqual "String foo" "foo" $ runId x
let test2 = TestCase $ assertEqual "String foobar" "foobar" $ runId $ idPlusBar x
-- Id Int tests
let y = Id 3 :: Id Int
let test3 = TestCase $ assertEqual "Int 3" 3 $ runId y
let test4 = TestCase $ assertEqual "Int 5" 5 $ runId $ idPlusTwo y
let test5 = TestCase $ assertEqual "Int 11" 11 $ runId $ idComposed y
-- run tests
let tests = TestList [ TestLabel "foo" test1
,TestLabel "foobar" test2
,TestLabel "int 3" test3
,TestLabel "int 5" test4
,TestLabel "int + *" test5
]
runTestTT tests
</code></pre>
<br />
So far, none of this is too challenging. You may have noticed that there's absolutely no point to any of it, however. That's OK! The idea was just to get used to wrapping stuff up and operating on it. The first version of "idPlusBar" is interesting because it illustrates how things work in "do-notation."<br />
<br />
If you've got this monad loaded up into ghci, you can see this, too:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> *Main> let x = Id "foo"
*Main> x
Id "foo"
*Main> idPlusBar x
Id "foobar"
*Main> let f x = x >>= \a -> Id (a ++ "bar")
*Main> f x
Id "foobar"
</code></pre>
<br />
where "f" doesn't use "do", but rather "bind" directly.<br />
<br />
Now, if you're still with me, we shall leave the tidy, simple world of identity, and enter the mysterious wonderment that is "state."<br />
<br />
The first part of this excercise involved coming up with a super simple case where I needed to have a state parameter of some kind and a bunch of functions that depended on it. I elected to make the State an "Int" and the value the functions operate on a List of Ints. The key is that in order to be sequenced correctly the state would have to be passed from one function to the next. The functions use the state int to transform the lists, and to add complication (and make it easy to prove it's working properly) I will increment the state with each function that gets executed.<br />
<br />
As you probably know, you can do it by "threading the state" and not using monads at all. Let's try that first:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> addS :: Int -> [Int] -> [Int]
addS s xs = map (+ s) xs
timesSminus1 :: Int -> [Int] -> [Int]
timesSminus1 s xs = map (* (s-1)) xs
addStimes2 :: Int -> [Int] -> [Int]
addStimes2 s xs = map (+ (s*2)) xs
</code></pre>
<br />
where s::Int is the 'state' and the list is the thing to be transformed<br />
<br />
Now we can sequence these functions, and "manually" increment the state:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> allS :: Int -> [Int] -> [Int]
allS s xs = let as = addS s xs;
s' = s+1;
as' = timesSminus1 s' as;
s'' = s'+1;
in addStimes2 s'' as'
</code></pre>
<br />
It's already easy to see, even with this simple example, that this has the potential to become quite cumbersome. It's also obvious that the incrementing of the state is going to get awfully repetitive and should be abstracted in some way.<br />
<br />
Enter the state monad...<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> type S = Int
newtype St s a = St { runState :: S -> (S, a) }
instance Monad (St s) where
return value = St $ \state -> (state, value)
monadSt >>= func = St $ \state->let (state', value) = runState monadSt state
in runState (func value) (state'+1)
</code></pre>
<br />
Note: this is not a generic state monad, but one which works exclusively on Ints and increments the state with each function execution (in "bind").<br />
<br />
Here are our functions again, using the monad:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> addS' :: [Int] -> St Int [Int]
addS' xs = St $ \state -> (state,map (+ state) xs)
timesSminus1' :: [Int] -> St Int [Int]
timesSminus1' xs = St $ \state -> (state,map (* (state-1)) xs)
addStimes2' :: [Int] -> St Int [Int]
addStimes2' xs = St $ \state -> (state,map (+ (state*2)) xs)
allS' :: [Int] -> St Int [Int]
allS' xs = do
as <- addS' xs
as' <- timesSminus1' as
as'' <- addStimes2' as'
return as''
</code></pre>
<br />
The last function in particular is quite clean compared to its non-monadic cousin.<br />
<br />
Once again, we'll use HUnit to test some assumptions...<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> main = do
putStrLn "running tests"
-- set up some values to use
let xs = [1,2,3]
let state1 = 3
let state2 = 4
let state3 = 5
-- threading state tests
let test1 = TestCase $ assertEqual "increment by s" [4,5,6] $ addS state1 xs
let test2 = TestCase $ assertEqual "times (s-1)" [3,6,9] $ timesSminus1 state2 xs
let test3 = TestCase $ assertEqual "increment by s*2" [11,12,13] $ addStimes2 state3 xs
-- put it together in allS
let test4 = TestCase $ assertEqual "the lot" [22,25,28] $ allS state1 xs
-- state monad tests
let test1' = TestCase $ assertEqual "m increment by s" [4,5,6] $ snd $ (runState $ addS' xs) state1
let test2' = TestCase $ assertEqual "m times (s-1)" [3,6,9] $ snd $ (runState $ timesSminus1' xs) state2
let test3' = TestCase $ assertEqual "m increment by s*2" [11,12,13] $ snd $ (runState $ addStimes2' xs) state3
-- monadic state passing in allS'
let test4' = TestCase $ assertEqual "m the lot" [22,25,28] $ snd $ (runState $ allS' xs) state1
-- run tests
let tests = TestList [ TestLabel "test1" test1
,TestLabel "test2" test2
,TestLabel "test3" test3
,TestLabel "test4" test4
,TestLabel "m test1'" test1'
,TestLabel "m test2'" test2'
,TestLabel "m test3'" test3'
,TestLabel "m test4'" test4'
]
runTestTT tests
</code></pre>
<br />
The real eye-opening thing here is that allS' doesn't take a state parameter at all. That's because allS' is returning a state monad (a function from state to state,value). So, we pass the state to the function (or value) which is returned by this function. This is the trick... and it is a little bit like magic!<br />
<br />
So,<br />
<br />
allS' takes the list ("xs") and returns the monad. runState extracts the function from the monad (\s -> (s,a)), and we pass the initial state to that. That in turn returns the state,value pair, so we take the "snd" value of the tuple.<br />
<br />
This is it again:<br />
<br />
<pre style="background-image: URL(http://q-media.com/images/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> snd $ (runState $ allS' xs) state1
</code></pre>
<br />
Nice, huh?<br />
<br />
So, our function doesn't have to mess around with state at all. It's all handled in the monad's bind function, including the incrementing of the state's value with each function call.<br />
<br />
Once again, I would just add that the most striking thing in all this is its simplicity (a trait I've come to expect from Haskell). It seems that monads weren't that complicated after all.<br />
<br />
Now, what are these Monad Transformer things?Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com2tag:blogger.com,1999:blog-4261954431421179908.post-88516563554546316362010-09-09T21:48:00.002+01:002012-09-23T12:47:48.738+01:00Automated Unit Testing with ScalaCheckIt just sounds like a great idea, doesn't it? <i>Automated</i> testing? Fantastic! I can see the tagline now: Say Goodbye to Technical Debt Forever!<br />
<br />
Before I begin, let's establish what we're talking about: What do we really mean when we say automated testing? In this case, what I <i>do not</i> mean is "test automation". I'm not talking about <i>running</i> tests automatically, by, say, using Continuous Integration. I'm talking about test <i>generation</i>. Test generation means, if you've got a function that operates on a particular input, you specify a "property" (or several properties) about that function, and then "generators" will give you variations on the input data to test all the edge cases, etc., and tell you whether the property holds. You can use default generators for known datatypes, or define your own.<br />
<br />
I've been interested in the idea of automated testing for some time, having come across <a href="http://en.wikipedia.org/wiki/QuickCheck">QuickCheck</a> (a Haskell library). Please see this quite interesting interview with <a href="http://www.infoq.com/interviews/john-hughes-fp">John Hughes</a> (one of QuickCheck's creators) on Functional Programming.<br />
<br />
I don't know Haskell very well, and have done more with Scala to date, so I thought I'd have a look at <a href="http://code.google.com/p/scalacheck/">ScalaCheck</a> to satisfy my curiosity. The following represents a quick walk through of the basics of ScalaCheck.<br />
<br />
Please note: I'm using Scala 2.7.7 and ScalaCheck 1.6, and I'll do all this via the interpreter.<br />
<br />
So, on the command line...<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala -cp scalacheck_2.7.7-1.6.jar
scala> import org.scalacheck.Prop
</code></pre>
<br />
Nice. I just generated a hundred tests. This TDD business is going to be a breeze! But what was really going on here?<br />
<br />
"forAll" is a method of the Prop ("property") object that returns a Prop instance. The parameter in this case is a function which is implicitly converted into a Prop. That Prop's "check" method is called and the result of the property check is displayed.<br />
<br />
What was the property under test? For any given two Ints, if they are added together, the result of that addition will be equal regardless of the order in which they are added.<br />
<br />
Admittedly, not very interesting, but nonetheless, this example illustrates a number of things, not least of which is that ScalaCheck generated, by way of its default Int generator (or Gen), 100 tests which all passed.<br />
<br />
Here's a longer way to write the same thing which may make some of this clear:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> val f = (a:Int, b:Int) => a+b == b+a
f: (Int, Int) => Boolean =
scala> val p = Prop.forAll(f)
p: org.scalacheck.Prop = Prop
scala> p.check
+ OK, passed 100 tests.
</code></pre>
<br />
Great.<br />
<br />
And you can easily combine multiple properties. Let's say you have 3 Props (p1,p2,p3) the first two of which are correct and the last is not:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> (p1 && p2).check
+ OK, passed 100 tests.
scala> (p1 && p3).check
! Falsified after 1 passed tests.
> ARG_0: T(8,true)
scala> (p1 || p3).check
+ OK, passed 100 tests.
</code></pre>
<br />
Now, if you're like me, and you really want to know what's going on, note that it is possible to "collect" the data that is generated for the tests.<br />
<br />
For example...<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> val p = Prop.forAll((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a })
p: org.scalacheck.Prop = Prop
</code></pre>
<br />
This is essentially the same thing, but notice the insertion of the call to "Prop.collect".<br />
<br />
Now when we check it, we get this...<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> p.check
+ OK, passed 100 tests.
> Collected test data:
4% (0,0)
1% (-22,38)
1% (4,13)
1% (50,30)
1% (-1,44)
1% (4,28)
1% (12,-16)
1% (-1,-16)
1% (-11,2147483647)
1% (72,-17)
1% (8,-1)
1% (-3,-77)
1% (-46,-19)
1% (-23,-1)
1% (-26,-22)
1% (-21,46)
1% (-48,48)
1% (1,-13)
1% (18,1)
1% (19,-1)
1% (34,-3)
1% (-1,-29)
1% (-58,-63)
1% (1,15)
1% (4,11)
1% (17,-22)
1% (-8,-8)
1% (3,-15)
1% (-9,-36)
1% (-92,78)
1% (28,-42)
1% (-18,0)
1% (3,1)
1% (4,2147483647)
1% (-2147483648,10)
1% (37,-2147483648)
1% (8,-2147483648)
1% (21,0)
1% (1,39)
1% (17,88)
1% (0,-48)
1% (38,-44)
1% (0,6)
1% (2147483647,29)
1% (-1,-24)
1% (-28,11)
1% (-4,46)
1% (28,-63)
1% (35,8)
1% (9,12)
1% (18,29)
1% (8,-18)
1% (-3,-1)
1% (0,14)
1% (47,1)
1% (4,3)
1% (-84,-7)
1% (13,31)
1% (-60,29)
1% (-41,-43)
1% (-75,67)
1% (-3,2)
1% (-73,-57)
1% (-10,-13)
1% (15,-3)
1% (2147483647,-25)
1% (8,0)
1% (-79,76)
1% (48,3)
1% (-5,9)
1% (4,0)
1% (-2147483648,8)
1% (43,-25)
1% (-46,54)
1% (-9,-2147483648)
1% (52,33)
1% (-10,4)
1% (36,-20)
1% (1,0)
1% (30,1)
1% (-2147483648,22)
1% (-27,-12)
1% (0,1)
1% (-58,2147483647)
1% (38,-27)
1% (-38,19)
1% (23,38)
1% (11,-15)
1% (-2147483648,17)
1% (-17,51)
1% (-36,-59)
1% (1,14)
1% (-3,-11)
1% (-31,29)
1% (-60,-44)
1% (2147483647,1)
1% (4,16)
</code></pre>
<br />
So, we can see what's at work behind ScalaCheck's default generator (Gen) for "Int". A lot of random numbers there. Edge cases, positive and negative, zero, etc. Thanks, ScalaCheck! You just saved me a lot of test writing!<br />
<br />
Let's look at a couple of things you can do with "Gen".<br />
<br />
First, there's a "choose" method to choose amongst options. Say, we only wanted to check our property with Ints between 1 and 100.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> Prop.forAll(Gen.choose(1,100),Gen.choose(1,100))((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a }).check
+ OK, passed 100 tests.
> Collected test data:
1% (32,48)
1% (80,30)
1% (26,31)
1% (15,91)
1% (39,3)
1% (12,68)
1% (93,71)
1% (8,22)
1% (44,58)
1% (4,73)
1% (76,96)
1% (30,98)
1% (55,100)
1% (76,28)
1% (93,13)
1% (46,47)
1% (78,56)
1% (77,30)
1% (69,17)
1% (29,81)
1% (11,39)
1% (40,3)
1% (17,56)
1% (2,81)
1% (31,2)
1% (42,7)
1% (47,22)
1% (48,17)
1% (79,33)
1% (48,40)
1% (52,83)
1% (95,50)
1% (63,30)
1% (21,96)
1% (29,30)
1% (72,33)
1% (70,56)
1% (24,79)
1% (68,16)
1% (89,7)
1% (82,21)
1% (13,35)
1% (73,88)
1% (99,25)
1% (79,8)
1% (52,84)
1% (92,32)
1% (90,77)
1% (88,60)
1% (51,4)
1% (32,93)
1% (62,90)
1% (9,14)
1% (40,36)
1% (78,80)
1% (84,5)
1% (31,63)
1% (7,14)
1% (48,88)
1% (92,21)
1% (31,52)
1% (58,2)
1% (69,82)
1% (84,7)
1% (19,69)
1% (48,94)
1% (50,16)
1% (59,52)
1% (63,63)
1% (55,40)
1% (50,39)
1% (30,26)
1% (15,61)
1% (85,30)
1% (93,90)
1% (19,49)
1% (5,61)
1% (28,56)
1% (41,60)
1% (88,82)
1% (40,62)
1% (96,48)
1% (85,96)
1% (88,7)
1% (1,91)
1% (14,82)
1% (1,56)
1% (24,98)
1% (86,41)
1% (77,65)
1% (4,38)
1% (78,77)
1% (4,62)
1% (84,36)
1% (90,33)
1% (12,67)
1% (45,45)
1% (54,75)
1% (38,94)
1% (14,91)
</code></pre>
<br />
Or, we can see that the range is inclusive.<br />
<br />
Or, we can define a Gen which is based on conditional statements using "suchThat".<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> val smallOdds = Gen.choose(1,100) suchThat (_ % 2 == 1)
smallOdds: org.scalacheck.Gen[Int] = Gen()
</code></pre>
<br />
And use that for both our Ints:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> Prop.forAll(smallOdds,smallOdds)((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a }).check
+ OK, passed 100 tests.
> Collected test data:
2% (19,75)
2% (53,55)
1% (35,47)
1% (15,75)
1% (13,41)
1% (51,99)
1% (39,23)
1% (97,35)
1% (47,99)
1% (79,87)
1% (95,55)
1% (45,49)
1% (3,61)
1% (73,93)
1% (13,5)
1% (45,45)
1% (31,51)
1% (61,9)
1% (55,55)
1% (25,47)
1% (99,21)
1% (65,29)
1% (47,49)
1% (47,89)
1% (95,39)
1% (43,73)
1% (39,49)
1% (41,17)
1% (89,25)
1% (25,39)
1% (37,27)
1% (17,49)
1% (77,37)
1% (11,9)
1% (15,13)
1% (37,47)
1% (93,77)
1% (5,75)
1% (19,87)
1% (35,39)
1% (3,21)
1% (65,53)
1% (73,83)
1% (39,63)
1% (31,53)
1% (69,31)
1% (99,65)
1% (9,97)
1% (55,57)
1% (83,71)
1% (41,35)
1% (59,69)
1% (3,55)
1% (85,9)
1% (87,71)
1% (33,33)
1% (9,83)
1% (59,55)
1% (19,51)
1% (49,45)
1% (95,61)
1% (45,21)
1% (57,61)
1% (87,89)
1% (35,71)
1% (85,89)
1% (61,91)
1% (47,5)
1% (3,69)
1% (3,17)
1% (39,85)
1% (95,89)
1% (87,11)
1% (85,49)
1% (55,79)
1% (7,97)
1% (93,29)
1% (37,61)
1% (51,19)
1% (67,53)
1% (71,9)
1% (57,11)
1% (41,85)
1% (31,27)
1% (31,57)
1% (87,49)
1% (69,53)
1% (59,77)
1% (23,87)
1% (29,21)
1% (11,59)
1% (69,69)
1% (77,79)
1% (85,65)
1% (31,31)
1% (97,15)
1% (11,15)
1% (75,85)
</code></pre>
<br />
Let's say you wanted to generate data for an arbitrary case class. ScalaCheck comes with an "Arbitrary" class to do just that:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> case class T(a:Int, b:Boolean)
defined class T
scala>import org.scalacheck.Arbitrary
import org.scalacheck.Arbitrary
scala> val genT = for { a <- Gen.choose(1,10); b <- Arbitrary.arbitrary[Boolean] } yield T(a,b)
genT: org.scalacheck.Gen[T] = Gen()
</code></pre>
<br />
So, our class should be comprised of an Int value 1 to 10 (inclusive) and a boolean.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> Prop.forAll(genT)((t:T) => Prop.collect(t) { t.a >= 1 && t.a <= 10 && (t.b == true || t.b == false) }).check
+ OK, passed 100 tests.
> Collected test data:
9% T(3,true)
8% T(10,true)
7% T(6,false)
7% T(2,true)
6% T(4,false)
6% T(7,false)
6% T(2,false)
6% T(1,true)
6% T(4,true)
5% T(5,false)
5% T(8,true)
5% T(8,false)
5% T(3,false)
4% T(5,true)
4% T(7,true)
3% T(10,false)
3% T(1,false)
3% T(9,false)
1% T(9,true)
1% T(6,true)
</code></pre>
<br />
Lovely.<br />
<br />
Another interesting feature is the ability to classify values that are chosen by the generator. For example...<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> Prop.forAll(genT)(t => Prop.classify({ t.b==true }, "b is true")(p)).check
+ OK, passed 100 tests.
> Collected test data:
60% b is true
</code></pre>
<br />
Note: "p" is a property which holds in this case.<br />
<br />
Labels:<br />
<br />
Let's say you have three properties you are testing, and two of the three are correct...<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> val p1 = Prop.forAll(genT)((t:T) => t.a >= 1)
p1: org.scalacheck.Prop = Prop
scala> p1.check
+ OK, passed 100 tests.
scala> val p2 = Prop.forAll(genT)((t:T) => t.a <= 10)
p2: org.scalacheck.Prop = Prop
scala> p2.check
+ OK, passed 100 tests.
scala> val p3 = Prop.forAll(genT)((t:T) => t.b == false)
p3: org.scalacheck.Prop = Prop
scala> p3.check
! Falsified after 2 passed tests.
> ARG_0: T(1,true)
</code></pre>
<br />
Run in combination, it's impossible to know which property failed:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> (p1 && p2 && p3).check
! Falsified after 0 passed tests.
> ARG_0: T(2,true)
</code></pre>
<br />
So, you can label the individual properties using a special ":|" operator:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> (p1 :| "a greater than or equal to 1" && p2 :| "a less than or equal to 10" && p3 :| "b always false").check
! Falsified after 0 passed tests.
> Labels of failing property:
b always false
> ARG_0: T(8,true)
</code></pre>
<br />
Lastly, another cool feature which is very "scala-ish": implicit generator definitions. Basically, ScalaCheck uses implicit definitions to decide which generator to use. By default, there are implicit generators for basic types (Int, String, etc., as seen above where no generator definition was necessary for Ints). However, here's how implicit def works for our case class (T):<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> scala> implicit def arbT: Arbitrary[T] = Arbitrary(genT)
arbT: org.scalacheck.Arbitrary[T]
scala> Prop.forAll((t:T) => t.a >= 1).check
+ OK, passed 100 tests.
</code></pre>
<br />
Don't blink or you'll miss the magic!<br />
<br />
This is all gone into in much more detail in the <a href="http://code.google.com/p/scalacheck/wiki/UserGuide">ScalaCheck User Guide</a>, but I hope this overview will prove helpful to some.Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com0tag:blogger.com,1999:blog-4261954431421179908.post-17508834722476647012010-08-21T14:45:00.004+01:002011-01-17T16:12:30.501+00:00Developing a Project Projection Chart in Excel When the Backlog Contains Sub-Projects (Agile/Scrum)With the keyword being "Excel". (Oh, it's a particular beast, alright.)<br />
<br />
Lest I be burned at the stake by a ferocious pack of Excelites, I begin with the following admission: I am a newbie when it comes to Excel. I write this blog partly to get constructive feedback (should anyone have any), and partly to help anyone who might struggle with a similar problem and who might receive some modicum of joy from this method.<br />
<br />
<br />
<b>The Probelm</b><br />
<br />
First, let me define a problem: You have a backlog that contains work which is categorized in some way. The categories might be "sub-projects", "feature sets", or similar. You want to get a picture of when a particuar category is going to be worked on, or when work overall on a particuar category is going to be done. Furthermore, you don't just want to know, you want to be able to make a big, visible chart out of the data.<br />
<br />
This excercise was in part inspired by this blog post: <a href="http://www.management30.com/posts/2010/7/23/is-the-gantt-chart-useless-in-agile-projects.html">Is the Gantt Chart Useless in Agile Projects?</a> written by Michael Cardus on <a href="http://www.noop.nl/">Jurgen Appelo's</a> Management 3.0 site. That post deals specifically with Gantt charts in Agile, and a Gantt chart wasn't exactly what we needed, but is quite similar. So the "literature", so to speak, on doing Gantt charts in Excel should prove useful.<br />
<br />
Let's say your backlog looks like this:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTUojjwE0pRXP2L0rYczxrS1Yrwn9pMumYyy_23KyEkCze39ovp0MnA2wdY5N7z_9Q62dkJRS4_RYvAblICw-F_Zcdwf1-qi9yj_pY3A2Gr-OVdFSyhyphenhyphenB_Ev_polKEzfq_ovTXcyx5aHU7/s1600/backlogshort.JPG" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5507851204561438610" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTUojjwE0pRXP2L0rYczxrS1Yrwn9pMumYyy_23KyEkCze39ovp0MnA2wdY5N7z_9Q62dkJRS4_RYvAblICw-F_Zcdwf1-qi9yj_pY3A2Gr-OVdFSyhyphenhyphenB_Ev_polKEzfq_ovTXcyx5aHU7/s320/backlogshort.JPG" style="cursor: hand; cursor: pointer; float: left; height: 306px; margin: 0 10px 10px 0; width: 320px;" /></a><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
So, we have four categories with various stories which have been put in a completely random (uh, I mean very sensible and thought-out) order by the Product Owner. Each story has a size. We also (and maybe not everyone does this) keep old backlog items around and mark as "done", so we'll need to take that into consideration.<br />
<br />
Let's also assume that using Excel is a requirement. Please don't question this requirement; I did, and my arse still hurts from the spanking.<br />
<br />
The first thing I did when trying to "solve" this problem was to try a load of various ways of summarizing the data and visualizing it with built in Excel chart tools. Once I managed to reinsert all the hair that I had pulled out of my head (actually, there's still a spot on top which isn't covered and I'm pretty sure that wasn't there before ;-), I eventually settled on this method: Convert the story points to days based on velocity, calculate the work activity on a per day basis, then use a stacked bar chart to visualize.<br />
<br />
Let's look at those steps individually:<br />
<br />
<br />
<b>Step 1: Points to Days</b><br />
<br />
Now, I know what you're thinking: story points are not a measure of time! Yes, but that doesn't mean to say that we're not allowed to use our average (and worst case) velocity to make loose predictions for release planning purposes. Work with me here, I'm pretty sure I'm not violating some law of Scrum physics that's going to open up a hole in the Points/Time Continuum.<br />
<br />
The way I did this conversion was to use a few hidden columns in the backlog to show "preceding points" for each story (i.e. story points up to this story in the backlog which were not yet complete), as well as "days before" and "days at completion" both based on velocity.<br />
<br />
The formula for getting the preceding points was a bit tricky. To sum the "undone" work in the backlog that precedes this story:<br />
<br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre-wrap;">{=SUM(IF(($E4<>"y")*($D4>0)*($E$4:$E$14<>"y")*(ROW($D$4:$D$14)<ROW()),$D$4:$D$14,0))}</span><br />
<br />
Where D is the Points column, and E is the Done? column.<br />
<br />
It's an array formula (a key fact: look up array formulas if you don't know about them, they're quite cool) that basically looks at the whole table to get points for stories where "Done?" is not "y" and the row number is less than the current row number. Looks easy in hindsight, doesn't it!<br />
<br />
Note that the "*" is how we "AND" conditions in an array formula. Excel's "AND" function won't work here.<br />
<br />
The next two calculations really are easy.<br />
<br />
Calculate the number of days until this story will start (based on current velocity):<br />
<br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre-wrap;">=($F4/($C$23/7))</span><br />
<br />
Calculate the number of days until this work will be complete (based on current velocity and story size):<br />
<br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre-wrap;">=IF($E4<>"y",($F4+$D4)/($C$23/7),0)</span><br />
<br />
<div>Where F is the "preceding points" column we calculted above, C23 is the average sprint velocity and 7 is the number of days per sprint.</div><br />
Now, with those columns unhidden, the backlog looks like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDHv8QlPY1T4KgiNt1_OB3STd8DtV2cpCIfQrEiiJ9BsFP9LlL6pDZRmnu2jVEJ3rteWjRAxoEjfZM-mDsmicNlM5enhon3y9TClssqXmAqLib0oyTVpiDyxQ_Bi5E-EBBQOKa04hsi2Xd/s1600/backlog.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDHv8QlPY1T4KgiNt1_OB3STd8DtV2cpCIfQrEiiJ9BsFP9LlL6pDZRmnu2jVEJ3rteWjRAxoEjfZM-mDsmicNlM5enhon3y9TClssqXmAqLib0oyTVpiDyxQ_Bi5E-EBBQOKa04hsi2Xd/s320/backlog.JPG" /></a></div><br />
So, with that chore out of the way, let's talk about what we want this graph to look like. I always find with Excel that it's easier to think about what data you need to create a workable chart first, and then format the data accordingly. It's too easy to disappear down a rabbit hole if you take what seems to be sensible data and try to make the chart fit *it*.<br />
<br />
Ultimately, this is what we end up with (what we are going for):<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-Yi6dM4UsaHL4LrnN9uZ2JIONXJsFymfxFNxkGSURErDiDUJiPo16Ik9ua-JCJnqjLsQBUMwd5tAfswohSoxurltSp4bxEKvqEGQWnaMQ7U1WLADIac1UTi3UkbduK5epkjZLUQRrwpcV/s1600/backlogchart2.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-Yi6dM4UsaHL4LrnN9uZ2JIONXJsFymfxFNxkGSURErDiDUJiPo16Ik9ua-JCJnqjLsQBUMwd5tAfswohSoxurltSp4bxEKvqEGQWnaMQ7U1WLADIac1UTi3UkbduK5epkjZLUQRrwpcV/s320/backlogchart2.JPG" /></a></div><br />
The Gantt chart literature made it quite clear that a stacked bar chart would be our friend. I played around a bit with line charts and scatter plots, but nothing worked quite right. One thing that slightly bugs me about Excel is not being able to do a chart which is categorical on both axes. If that were possible then a mapping of our categories to a Sprint date (or Sprint number) would be relatively easy. I'm sure there's a mathematical reason for it (surely there's a reason for everything Microsoft does?), but Excel only supports value/value (scatter plot) and category/value (everything else).<br />
<br />
<b>Step 2: The Chart Data</b><br />
<br />
Now this is where it gets ugly. How the hell are we going to get a chart that looks like that from our data? Turns out, it's pretty easy if we just stop thinking about it and lay the data out in the dumbest way we can think of! (I've always had this theory that computers are really, really dumb, and programmers are just people who are good at breaking stuff down into simple instructions that make sense to a dummy!)<br />
<br />
Take a look at the picture below to see how this was done. Better yet, download the spreadsheet <a href="http://www.q-media.com/files/ProductBacklogChartTest.xlsx.zip">here</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh10_IGrrTGGPoE7vPOgZ3vpLqWZw2IkuHYI9NBohANEkaxVGDTwhkPJTYm0xVDSHAWy5bS0XIkzOFTq5_HG13J6VYpI1xyUvgIwLZWAAEvhIiyLbqFQPObQX4nQSz2kGFkFqIwZl11d3tT/s1600/backlogchartdata.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh10_IGrrTGGPoE7vPOgZ3vpLqWZw2IkuHYI9NBohANEkaxVGDTwhkPJTYm0xVDSHAWy5bS0XIkzOFTq5_HG13J6VYpI1xyUvgIwLZWAAEvhIiyLbqFQPObQX4nQSz2kGFkFqIwZl11d3tT/s320/backlogchartdata.JPG" /></a></div><br />
Note: the pic doesn't show the whole thing. There is data for every day that we want to show (42 days, or 6 sprints of 7 days, in this case).<br />
<br />
So, literally, there's a column for every day, and there's a cell for every category/day that has a 1 or 0 depending on whether there is work in that category for that day. The formula looks like this:<br />
<br />
Determine if this Category on this Day contains work (show 1 or 0):<br />
<br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre-wrap;">{=SUM(IF((Table1[category]=$U5)*(Table1[done?]<>"y")*(V$3>=Table1[days to start])*(V$3<Table1[days to finish]),1,0))}</span><br />
<br />
Where U is the Category column, and V3 points to the day number of the current column.<br />
<br />
So, basically, for all items in this category: is this day in between the start and finish day of these pieces of work?<br />
<br />
The "1" actually creates the "height" for our bar in the graph. The trick here is the use of the inverse items (blanks) to fill in the vertical space below other categories. The "blanks" are just set to "No Fill" in the chart, so they are literally blanks.Figuring that one out was my proudest moment using Excel to date (OK, you caught me, I'm easily impressed).<br />
<br />
A few other bits of trickery: eliminate gaps in bars so they look continuous; set horizontal axis major unit to 7 for sprint dates; delete left axis; delete "blanks" from legend; and we're done!<br />
<br />
So, please download that spreadsheet and use this if you like it. Or, if you have a better idea, I'd love to hear it!<br />
<br />
'Bye for now...Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com0tag:blogger.com,1999:blog-4261954431421179908.post-17869863451820951662010-06-08T22:52:00.015+01:002012-12-10T17:29:08.442+00:00What's So Funny About ATDD, BDD, and Dependency Injection?With apologies to Elvis Costello...<br />
<br />
So, I've been thinking about a way to introduce the concepts of Acceptance Test Driven Development (ATDD) and Behavior Driven Development (BDD) to our developers. I'd also like to cover Dependency Injection as this seems to be something that comes up a lot when we discuss TDD in our little neck of the woods.<br />
<br />
As an example, I've used an abstraction from a large project that we've been working on. The project is a data conversion project. I won't go into details, but the gist is this: take some data (probably from a database), perform some validation checks on it, filter stuff that we don't want, convert some stuff that we do want, and send it to some kind of output stream.<br />
<br />
Our project is Java-based, but I've chosen to use Scala for this example for a number of reasons:<br />
1. Scala seemed interesting, and I wanted to learn more.<br />
2. Scala has parser combinators which is perfect to test my idea of using a DSL to define our conversion rules.<br />
3. ScalaTest supports ATDD and BDD beautifully via its FeatureSpec and FlatSpec traits, respectively.<br />
4. Scala has some neat ways of dealing with dependency injection (see Chapter 27 of the Odersky book).<br />
5. Scala is fully interoperable with Java, so we can reuse any libraries or code from the existing project.<br />
<br />
See Scala and ScalaTest <a href="http://www.scala-lang.org/">here</a> and <a href="http://www.scalatest.org/">here</a>, respectively.<br />
<br />
So, let's get started... (please bear in mind that I am well aware that my Scala code, being a relative newcomer to it, is likely to be a bit rough... I welcome suggestions!)<br />
<br />
Firstly, I set up an ant build.xml that would compile my Scala code and run my ScalaTest tests. There are basically 3 files (not counting the build.xml): 1) test/UnitTests.scala, 2) test/AcceptanceTests.scala, and 3) src/Conversion.scala. It's a pretty small project for purposes of this exercise, so there's no need to get crazy here, plus it's nice to be able to easily ":load" files in the Scala interpreter (note: by the end it was just approaching large enough to warrant breaking some stuff out into more files, but I chose not to go there).<br />
<br />
Note: the code for this project can be downloaded <a href="https://github.com/sqpierce/conversion">here</a>. Please take a look. I am just going to focus this blog on a few of the more interesting points.<br />
<br />
The application design was set up with the idea of ATDD in mind. There would be a Conversion class that would take all the necessary bits as parameters: those bits being the datasource, the rules for the conversions, and some kind of output stream device. So, let's look at some of the acceptance tests first.<br />
<br />
The first thing to bear in mind was that I was going to need some kind of Data type for retrieving and carrying the input data. I knew that this might come from a database, but I wanted to be able to "inject" it for purposes of these tests. I decided to go with a simple Map[String,String] to capture field names and values. There would be an abstract class called "Data" and one called "DataSet" that I could implement to get my tests set up. The DataSet class would have a foreach method, so that rows could be read from a database one at a time for processing (rather than pass a whole set of data at once, as our datasets are quite large in reality).<br />
<br />
Additionally, I would need "Rules". I figured that these would be read from a file, and would comprise our "DSL" bit, so I opted to have these be simple Strings. Again, an abstract class called RuleSet (List[String] for testing) would also have the potential for some other functionality, such as reading the Rules from a file.<br />
<br />
I also used a static "Log" object to capture system output for monitoring. The default Log would just write to STDOUT, but I also implemented a version that would write to a list buffer, so that I could capture that stuff for testing, too.<br />
<br />
Also, an Output class which I implemented as a ListBuffer as well for testing. In reality, this Output class could write to a database, or send data across the network, etc.<br />
<br />
Lastly, a Conversion object which takes each of our little dependencies via its apply method, and does it's magic (basically apply all Rules to each Data object in the DataSet and capturing the output via our Output object).<br />
<br />
Here's a snapshot of the acceptance tests...<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> class RuleApiFeatureSpec extends FeatureSpec with GivenWhenThen with MustMatchers {
val d1 = MapData(Map(("id","1"),("a","5foo"),("b","1234bar234")))
val d2 = MapData(Map(("id","2"),("a","foo"),("b","bar45")))
val d3 = MapData(Map(("id","3"),("a","454foo45"),("b","445bar")))
val d4 = MapData(Map(("id","4"),("a","43foo45"),("b","bar87")))
val d5 = MapData(Map(("id","5"),("a","foo45"),("b","3bar4546")))
val d6 = MapData(Map(("id","6"),("a","cfoo"),("b","bar")))
object MapDataSet extends DataSet {
// in production, DataSet may come from a database (can be read one record at a time in foreach)
val data = List(d1,d2,d3,d4,d5,d6)
def foreach(f:(Data)=>Unit):Unit = { data.foreach(f) }
}
val r0 = "f:b /^[0-9][0-9]+.*/ Convert x:transformAandB" // convert a and b if b starts with two numbers
val r1 = "f:b /^[0-9]+.*/ Convert x:transformA" // convert a if b starts with a number
val r2 = "f:a /^[0-9]+.*/ Filter" // filter if a starts with a number
val r3 = "f:b /^b.*/ Convert x:transformB" // convert b if b starts with "b"
val r4 = "f:a /^c.*/ NoMatch" // bad rule
val rules = RuleList(List(r0,r1,r2,r3,r4),MyRuleParser)
class ListOut extends Output {
var container = new ListBuffer[Data]()
def apply(dataIn:Data) = {
container+dataIn
}
def getById(n:String):Data = { container.filter( (d) => d("id") == n )(0) } // assuming one match here
override def toString = "***OUTPUT***\n"+container.mkString("\n")
}
Log.setType("list")
feature("Conversion API"){
scenario("Filtered Output"){
given("The default testing scenario set up above")
when("I run the conversion")
val out = new ListOut
Conversion(MapDataSet,rules,out)
then("I expect to see five records in the output")
out.container must have length 5
then("The filtered record should be id 4")
out.container.map( d => d("id") ).contains("4") must be === false
then("The records not filtered would be 1,2,3,5,6")
out.container.map( d => d("id") ).sameElements(List("1","2","3","5","6")) must be === true
}
}
}
</code></pre>
<br />
So, this system is geared toward Acceptance Testing, via dependency injection. And ScalaTest's FeatureSpec is great for this. It even gives you a nice "Given When Then" syntax, so that your real world acceptance criteria can easily be translated into automated acceptance tests. You are writing acceptance criteria, aren't you!?!? ;-)<br />
<br />
Other acceptance tests follow more or less the same pattern.<br />
<br />
BTW, Here's the code for the Conversion object:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> object Conversion extends Logging {
def apply(data:DataSet, rules:RuleSet, output:Output) = {
data.foreach(
d => {
log("PRE PROCESSING: "+d)
val o = rules(d)
log("POST PROCESSING: "+o)
if(!o.isFiltered()) { log("ADDING TO OUTPUT: "+o); output(o); }
}
)
}
}
</code></pre>
<br />
The foreach method on the DataSet is called and the rules applied to each resulting object. So, by implementing a "DbDataSet", for example, these records could be pulled from a database.<br />
<br />
I'll just show the RuleSet code, because I love this little Scala trick:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> case class RuleList(rs:List[String],rp:RuleParser) extends RuleSet with Logging {
val rules = rs.map { r => rp.parse(r) }.filter{ r => r != None }.map{ r => r.get }.toSeq
def apply(dataIn:Data):Data = {
log("APPLYING RULES")
Function.chain(rules)(dataIn)
}
}
</code></pre>
<br />
I suspect there might be a cleaner way to do that map/filter/map thing, but the cool thing is the line: Function.chain(rules)(dataIn). Basically, with the "rules" being implementations of Function[Data,Data], it's basically saying "apply this data object to this chain of Rule functions". Nice.<br />
<br />
This line sets up the rules, and references the RuleParser:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> val rules = RuleList(List(r0,r1,r2,r3,r4),MyRuleParser)
</code></pre>
<br />
Here's the code for the RuleParser:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> object MyRuleParser extends RegexParsers with RuleParser {
def fieldStart:Parser[String] = "f:"
def field:Parser[String] = "[A-Za-z]+".r ^^ { _.toString }
def functionStart:Parser[String] = "x:"
def function:Parser[String] = "[A-Za-z]+".r ^^ { _.toString }
def pattern:Parser[Regex] = "/.*?/".r ^^ { case p => new Regex("/".r.replaceAllIn(p,"")) }
def action:Parser[String] = "Filter|Convert".r ^^ { _.toString }
def rule1:Parser[Rule] = fieldStart~field~pattern~action~functionStart~function ^^ { case fs~f~p~a~xs~x => Rule(new PatternMatcher(f,p),a,ConvertFuns.functions(x)) }
def rule2:Parser[Rule] = fieldStart~field~pattern~action ^^ { case fs~f~p~a => Rule(new PatternMatcher(f,p),a,NoConvert) }
def rule:Parser[Rule] = rule1 | rule2
def parse(input:String):Option[Rule] = parseAll(rule,input) match {
case Success(e,_) => Some(e)
case f: NoSuccess => None
}
}
</code></pre>
<br />
This uses parser combinators to turn our "Rule" (strings) into "Rule" objects. Rule itself looks like this:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> case class Rule(matcher:MatchFun, action:String, conversion:ConvertFun) extends Function[Data,Data] with Logging {
def apply(dataIn:Data):Data = {
log("RULE: "+this.action+" on data "+dataIn+":")
if(dataIn.isFiltered){ log("FILTERED"); dataIn }
else if(matcher(dataIn)){
action match {
case "Filter" => { log("FILTERING"); dataIn.setFiltered() }
case "Convert" => { log("CONVERTING"); conversion(dataIn) }
case _ => { log("BAD ACTION"); dataIn }
}
}
else{ log("NO MATCH"); dataIn }
}
}
</code></pre>
<br />
Now, I'd like to talk about unit tests.<br />
<br />
For unit tests, I chose to use the FlatSpec trait in ScalaTest. The reason is because I wanted to do BDD, with the basic "Object when something should something in...{ code }" type syntax, but I really didn't like all the nesting that goes on in most libraries. Bill Venners has solved that problem for us with FlatSpec. The name is as it implies. Here are some examples...<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> class MapDataSpec extends FlatSpec {
val m = MapData(Map(("id","1")))
"A MapData object with an 'id' field" should "have an 'id' key" in {
expect(true){ m.has("id") }
}
"A MapData object with an 'id' field with value '1'" should "return that value for that key" in {
expect("1"){ m("id") }
}
"A MapData object which has no 'a' field set" should "throw error when accessed" in {
expect(false){ m.has("a") }
intercept[NoSuchElementException]{ m("a") }
}
"A MapData object which has a field set" should "recognize that key and return that value for that key" in {
val m2 = m.set("a", "foo")
expect(true){ m2.has("a") }
expect("foo"){ m2("a") }
}
"A MapData object which has not had filtered set" should "return false for isFiltered" in {
expect(false){ m.isFiltered }
}
"A MapData object which has filtered set" should "return true for isFiltered" in {
val m2 = m.setFiltered
expect(true){ m2.isFiltered }
}
}
</code></pre>
<br />
And the unit tests continue on like that (please see linked code for details). Nice, simple syntax. Very intuitive.<br />
<br />
I hope this makes some sense, and proves helpful to some.<br />
<br />
Edit: BTW, I have found a slightly more elegant way of doing this:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> val rules = rs.map { r => rp.parse(r) }.filter{ r => r != None }.map{ r => r.get }.toSeq
</code></pre>
<br />
which is this:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> val rules = rs.map { r => rp.parse(r) }.flatMap{ x=>x }.toSeq
</code></pre>
Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com0tag:blogger.com,1999:blog-4261954431421179908.post-71363668475449806522010-03-09T02:03:00.006+00:002018-09-20T15:57:52.806+01:00On the Fence: API vs DSL for programming interactive music in the DOME<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br />
</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">I've been working on an "online" version of the <b>DOME</b>, which is an interactive music engine for video games that I've been developing on and off for a number of years (for more info see <a href="http://www.dometechnics.com/">Dometechnics)</a>. By "online" I mean a version of the engine that could be embedded in a web page, so that it could be made available for developers of Flash and Javascript games.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">The engine itself is a Java applet, which is scriptable via Javascript. Java, for all its faults, is still the best platform for producing MIDI-based synthesized music in a web page. Thanks for nothin', Adobe. Keeping an eye on Google's <a href="http://code.google.com/p/nativeclient/">NativeClient</a>, though.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">This blog post will take a look at the challenges of getting Javascript and Java to talk to one another in a web page, but mostly, I want to look at the design decisions that I've been making about the API for doing music programming and whether or not it would be more appropriate to call it a "DSL" (although the distinction might just be syntactic sugar).</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">In order to make sense of this post, I should explain a little bit about how the DOME works. It's based on the idea of creating a "score" in the sense of a theme and variations on that theme, followed perhaps by another theme. Loosely, it's designed to enable "cinema-like" scores in interactive contexts: "Films often have different themes for important characters, events, ideas or objects, taking the idea from Wagner's use of leitmotif. These may be played in different variations depending on the situation they represent, scattered amongst incidental music." (<a href="http://en.wikipedia.org/wiki/Film_score">source Wikipedia</a>).</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">It sort of looks like this (from Javascript):</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span></div>
<div>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> dome.setTheme( _JSON-formatted-theme-data_ );
dome.play();
var minor_variation = 'E-&gt;Eb,B-&gt;Bb|!bar:4'; // C major to C minor, but not in bar 4
dome.bind( _some-dom-element_, 'onclick', function(){ dome.vary(minor_variation) } );
</code></pre>
</div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">See how easy!?!</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">The interesting part in terms of this API vs DSL discussion is the line where the variation is defined. Because of limitations in passing anything other than strings or numbers from Javascript to Java, we might have ended up with something like this:</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span></div>
<div>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> dome.vary('pitch', '+', 5); // not a very attractive looking command to use (and operators as strings? ...bleh)
</code></pre>
</div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">Instead, we can do this:</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span></div>
<div>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> dome.vary('pitch+5'); // better. now it's like a mini-script (will require some parsing magic, though)
</code></pre>
</div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">...Great. (gratuitous <a href="http://www.youtube.com/watch?v=TebUMhJAKSM">Fast Show</a> reference)</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">That's all well and good, but in fact, the API approach works better in certain situations. Namely when there's a lot of data to pass (for instance, when the initial theme is established), because an API call can just take a JSON-ified Javascript object as a string. That's because we can take advantage of existing libraries for parsing JSON. But having a DSL for creating variations, in this case, allows for the music programming to occur in a more natural-language type of way.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">"The key point is that at each layer the API/DSL should allow the user ... to express the intent of what they want to do as easily as possible." (<a href="http://www.testdrivensoftware.com/?p=85">http://www.testdrivensoftware.com/?p=85</a>)</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">As a side note, if you're the type of person who gets a kick out of making fun of DSL fan boys, I would direct you to chromatic's <a href="http://www.oreillynet.com/onlamp/blog/2007/05/the_is_it_a_dsl_or_an_api_ten.html">amusing discourse</a> on the topic.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">So, I've got an API to dump a Javascript object on Java and a DSL for scripting that object, once it's deserialized and appropriately instantiated in Java. In a sense this is still API, because it's essentially a function call on an object, passing in a string. There is no interpreter in the traditional sense of a scripting language. But because the string format for defining variations is a mini-language, it kind of becomes a DSL.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">...Great.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">Another option might have been to use Java 1.6's scripting API (<a href="http://java.sun.com/developer/technicalArticles/J2SE/Desktop/scripting/">javax.script</a>), but I had made the decision early on that I wanted this system to be compatible with older versions of Java. I believe the difference is that with Java 1.6, one could theoretically write Javascript code and have that evaluated in the Java application (thereby allowing Javascript functions to manipulate the Java objects). It's a nice idea, but, not critical, and Java 1.5 (and previous) compatibility seemed more important. Why? Because at the time of this writing, the MIDI functionality in Java 1.6 on Mac OS X is broken... that's good enough for me!</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">Another fun point I'd like to make: At one point during the development of this API/DSL, I fell afoul of a little difference between scripting an applet via the Rhino shell and doing it in a web page. In Rhino, you can return a Java object to Javascript from a method in the applet, and pass that object back to the applet in another function call. Not so via <a href="http://en.wikipedia.org/wiki/LiveConnect">LiveConnect</a>!</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">For example:</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span></div>
<div>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> var applet = document.MyApplet;
var obj = applet.getSomeObject(); // return a Java object
applet.doSomethingWithObject(obj); // OK in Rhino! Will cause 40 days and nights of floods if you try this in a browser!
</code></pre>
</div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">So, one kind of work around, is to have Java use a method on the object to execute the relevant code. This could involve having to share state between the applet and the object in question.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span></div>
<div>
</div>
<div>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUZe6U-J3LaBc4d0gDG2nsd0OeqgiJ3-_siV_9eVAIB23Ak8ZQD6siGdbfAOThrRmnDaFTUNQ0f7pkFZSBFMr7vezNofvcQtky5X9j8MSOG4i1FX1aiMz9CuexEP-lSoStN2E-4Q06CKgv/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> var applet = document.MyApplet;
var obj = applet.getSomeObject(); // return a Java object
obj.doSomethingWithMe(); // works OK in a browser
</code></pre>
</div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">...Great.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br />
</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">So, let's summarize: Embed the DOME in a web page as an applet and make it "scriptable" via Javascript, and support Java 1.5 for unbroken MIDI. With the goal being the ability to create a theme and define variations on that theme, and invoke those variations by binding them to Javascript events (hence making interactive music possible in Flash and Javascript based games). Done! Alright, not so much done, but it is a working prototype.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;">Because this is not really ready for prime time, I have produced the following screencast to demonstrate the ideas discussed in this blog. I am actively looking for development resource and industry contacts to bring this technology to fruition. Please come back to follow further progress.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS',sans-serif;"><br />
</span></div>
Steve Piercehttp://www.blogger.com/profile/05111167194178030567noreply@blogger.com0