Q-media

Fun with Financial Data (or As Much Fun As You Can Have Without Actually Making Any Money)

2011-02-06T19:26:00.005Z

I've started reading the book Data Analysis with Open Source Tools by Philipp K. Janert. It's very well written and informative, and I highly recommend it. I've only scratched the surface so far, but I decided that I wanted to try to apply some ideas from it to a problem that had been kicking around in my head for while.

And then I figured: why not write a blog post about it! (After all, no one reads this thing anyway ;-)

This problem occurred to me a while back while reading another O'Reilly book: Programming Collective Intelligence by Toby Segaran. The idea involves looking at a couple of pieces of information that are available on the web: Financial price data and financial news (both of which are freely available at Yahoo! Finance). I realize that the data that Yahoo publishes is not particularly real-time and one wouldn't probably want to build an HFT system on it, but it will serve to do a little studying.

I didn't really have a hypothesis in mind when I started. Taking a cue from Mr. Janert, I decided to take a look at the data with an open mind (no pre-conceived notions). At the heart of this inquiry, however, is a two-tier question:

Is there some kind of correlation between when news gets published online and when stock prices move? and
Is there a predictable sequential relationship that can be construed?

Why do this? Well, it's interesting. I'd been interested for a while in finding a worthwhile excuse to do something with financial data. Besides, there's a small chance that it might reveal an interesting opportunity (if news precedes predictable changes in price? Cha-ching!). More likely, however, I was expecting that I'd find that We, The Sheeple are trailing the HFT systems and insiders and there is little or no hope for us little folks to make a thin dime playing the market.

Please note that this is a first pass at this. By no means am I imagining that this is a comprehensive approach. Indeed, I would be very grateful if anyone has any ideas for what to look at next (please comment!).

For those who like the punchline first, here's the spoiler: the evidence is inconclusive as to whether any predictions can be made in price change based on headlines. Further study might reveal something, but at first glance, it just doesn't add up (pun intended).

Conversely, and somewhat surprisingly, it is not conclusive that price changes precede news. In other words, it is not clear cut that the news follows the market.

However, there is clearly a correlation in the data: big price shifts, high volume, and an increase in published news tend to go hand-in-hand. This is an unremarkable finding, but nonetheless, it was an interesting exercise.

Now, despite having warned you that these findings are dull as dishwater, if you are really interested, I presume you are still with me...

To analyze the problem, the first thing I did was to set up a method for capturing data. Once I had a bit of data to look at, then I could begin my inquiry.

I don't really want to spend a lot of time discussing the specific technologies employed, as that's not really the point of this post, but I will just summarize this step briefly:

A couple of shell scripts employing curl (via cron) to grab data at reasonable intervals: every minute for price data, and every hour for news data.
A couple of Scala scripts for parsing that data and dumping into a sqlite database.
SQL queries to grab the data (grouped as appropriate) in CSV format to pull into Excel (where I can create charts, where I can create charts for visualization, etc.)

As you can see, very low-tech, indeed. All in the spirit of keeping it simple. I thought about using matplotlib, which is a great Python-based tool for doing data analysis that I've had the pleasure of using a number of times in the past. But, as the data set I was focusing on was pretty small, Excel did the job nicely.

In another iteration of this inquiry, I might be inclined to try out R or Octave, and dig a little deeper. Perhaps I'll even take a further look at what data is available and try to expand the reach. If I'm feeling especially crazy, I might even give Hadoop a try.

For purposes of this exercise, I will be looking at this data over the course of a single trading week: January 10-14 2011 (inclusive), and I'm just grabbing data on stocks that make up the Dow Jones Industrial Average. No particular reason other than that that was a convenient week to do it, and it seemed like a sufficient amount of time for this initial undertaking.

I figured I would take a "big picture" view first. See whether just glancing at the data indicates anything interesting. If something jumps out, then drill down further and see where it takes me.

To put this in context, here's a graph showing the Dow's movement around that time:

Prices had been coming down the previous week, but were poised to generally go up this week (hindsight being 20/20 and all).

The first things I want to look at are:

How many news headlines are there per stock per day? (And look for spikes in these numbers.)
Significant shifts in prices.
Significant shifts in volumes.

That alone should establish where there's a relationship that's worth exploring. Doing this on a per stock per day over these few days should be sufficient to start. Then, if motivated, look at the intra-day prices and specific publish times of articles to see if one can determine whether it's the chicken or the egg that comes first.

To start, here's some summary information about our headline data per stock per day (biggest ranges, and largest gaps between mean and max are shown highlighted):

And here it is again, measuring number of standard deviations from the mean (largest numbers highlighted):

Those spikes are a lot clearer when looking at them charted:

Looking at these, just based on spikes in number of news articles, I'd be interested in looking further at the following:

(Because I know what headlines at Yahoo tend to get grouped with stocks in the same industries, I'm looking for those relationships as well.)

DuPont (DD), Johnson & Johnson (JNJ), and Procter & Gamble (PG) on Monday.
AT&T (T) and Verizon (VZ) on Tuesday.
Merck (MRK) and Pfizer (PFE) on Thursday.
Bank of America (BAC) and JP Morgan (JPM) on Friday.

Now, let's see if that jives with what we can see in price data (note: this is shown by price range for the day as a fluctuation (max-min) percentage from the previous day's close):

Looking at the price data, there is certainly enough coming through to believe that there is a correlation here. Much of the data points to the same stocks/days. I might be inclined to filter my list, however, as it least some of the headline data is likely to be associated with certain stocks only because of the industry relationship:

DuPont (DD) on Monday.
AT&T (T) and Verizon (VZ) on Tuesday.
Merck (MRK) on Thursday.
Bank of America (BAC) and JP Morgan (JPM) on Friday.

That reduces it to six different stocks on four different days, which seems like a reasonable number to analyze further (with two industry relationships to look at).

I might also be inclined to add some based on the price data alone, like McDonald's (MCD) or American Express (AXP) on Friday, but when looking at the headline data, there's not really enough to go on. (Most of the headlines for McDonald's on the Friday were actually about Starbucks.)

And, just to complete the picture, I'll have a look at volumes as well.

Again, a lot of the same players emerge, which leads me to think that I'm on the right track. The correlation is clear. Now, I'll have to look at the intra-day data to see if one can draw any conclusions moment-to-moment on the specific scenarios that I've identified.

DuPont (DD): Monday January 10th, 2011

First off, the data... (Note that the price column is price range during that hour as a percentage)

The following chart shows the deltas in headline count, price, and volume as a percentage of the total change over the course of the day (this allows unlike numbers to be compared in like manner). In particular, note that I've summed the delta of pre-open headlines into the first hour's number, which might exaggerate it's effect on the chart. But looking at the data you can certainly see that there was significant activity pre-open.

And, price for the day...

One thing's for certain, there was definitely a spike in the number of headlines at 11 am which followed the steep change in price. But most of the headlines from that day were before the bell, and the big dip in price happened before 10 am. By 11 am the price was fairly stable.

So, what were the headlines? Well, here are all the headlines that were published before 11 am:

07:07 "Duke, Progress Energy, Sara Lee, Strayer, WellPoint: U.S. Equity Preview"
07:15 "US STOCKS-Futures lower on Portugal concerns despite M&A"
07:24 "CNNMoney Pre-Market Report - Jan. 10, 2011"
07:25 "DuPont to Buy Danisco for $5.8 Billion"
07:25 "Futures Fall; DuPont, Earnings in Focus"
07:32 "Disappointing Jobs Data Weighs on Futures"
07:45 "[$$] Danisco to Recommend DuPont Bid"
07:56 "US STOCKS-Futures lower as Portugal concerns offset M&A"
08:08 "Let's Make a Deal"
08:19 "Futures follow Asia, Europe lower"
08:27 "UPDATE 1-Goldman Sachs raises US chemicals sector to attractive"
08:30 "[$$] The Day Ahead: Merger Monday"
08:40 "UPDATE 3-DuPont's Danisco bid sends sector shares higher"
08:53 "Before the Bell: Duke Energy, DuPont, Verizon in spotlight"
08:59 "U.S. Stocks Poised For Weak Open"
09:09 "[video] News Hub: Before the Opening Bell [1.4 min]"
09:14 "[video] Faber Report"
09:20 "Danish currency firms on Danisco takeover"
09:40 "CNNMoney Market Report - Jan. 10, 2011"
09:44 "Dow Falls at Open Despite Rise in M&A"
09:47 "Verizon iPhone Announcement Seen for Tuesday"
09:48 "Stocks Dip on Euro Debt Concerns"
09:50 "Portugal Bailout Push Mars Merger Monday"
09:59 "[audio] Opening Bell from MarketWatch Radio Network [1.0 min]"
10:00 "Monday Morning Quarterback: Mergers, Foreclosure and Hank Moody"
10:16 "Wall St down as Portugal woes offset M&A deals"
10:39 "US equities fall on eurozone fears"
10:57 "Stocks to Watch: Union Pacific, Exxon and More ..."

Clearly the buyout of Danisco was a big factor here, but I'm having a hard time understanding why it had the effect it did on price. It almost seems counter-intuitive that the share price should fall so dramatically in the first 20 minutes of trading, given the seemingly "good news" of this buyout. Perhaps it's something to do with exchange rates and the European problems that were weighing on the market? Was it this: Danish currency firms on Danisco takeover? (09:20)

Interesting, but by no means conclusive.

One headline in there jumps out, however, and makes a nice segue:

09:47 "Verizon iPhone Announcement Seen for Tuesday"

That'll explain our next subject: Verizon and AT&T activity on Tuesday.

Verizon (VZ): Tuesday January 11th, 2011

The spike in price at around 11 am when all those articles were published (which corresponded to the time of the Verizon iPhone launch) is certainly interesting. But I don't think it would take a genius to figure out that that might happen.

The other scenarios below also confirm what was stated previously, so I'm happy to leave the rest of this as an exercise for the reader. There is a correlation between headline count and price changes, but not a clearly sequential one. Again, I welcome feedback or ideas. I hope to come back to this at some stage and delve a little deeper. In the meantime, here's the rest of that data, starting with the Verizon headlines...

00:01 "IPhone May Cost Verizon Wireless $5 Billion in Subsidies in First Year"
00:15 "ANALYSIS - AT&T faces tough year after losing iPhone exclusive"
02:47 "Telecoms' Dividend Yields"
03:03 "With iPhone Taunts, AT&T-Verizon Rivalry Escalates"
03:51 "11 High Yield Stocks to Consider Now"
05:00 "[video] Verizon iPhone Data Plan"
06:00 "Nielsen: Young People Across The Globe Love Their Cell Phones (But Use Them Differently)"
06:45 "Half of Dow Companies to Post Profit Gains"
06:50 "Ahead of the Bell: Verizon iPhone"
07:27 "10 Things You Need To Know Before The Opening Bell"
07:27 "10 Things You Need To Know Before The Opening Bell"
07:30 "CNNMoney Pre-Market Report - Jan. 11, 2011"
07:32 "Lowenstein Says Verizon's Network Ready to Handle IPhone: Video"
07:39 "10 Things You Need To Know This Morning"
07:40 "iPhone Raises Stakes of AT&T-Verizon Rivalry"
07:45 "Verizon big winner from having iPhone? Not so fast"
07:58 "AT&T and Verizon Trade Taunts Over iPhone"
08:03 "[video] 3 Stocks I Saw on TV"
08:22 "Will Verizon Demolish AT&T? It Did So Long Ago"
08:24 "The Big Questions About Verizon's IPhone"
08:24 "Verizon big winner from having iPhone? Not so fast"
08:25 "McCourt Says Verizon's Network Can Handle Apple's IPhone: Video"
08:36 "How Important Is Offering the iPhone on Verizon to Apple?"
08:45 "[$$] ETF Play of the Day: XLK"
08:56 "Liveblog: Verizon set to launch the iPhone. Finally."
09:02 "[video] AM Report: Chinese Stealth Jet Takes Flight [8.0 min]"
09:17 "[video] 3 Networking Stocks Set for Cloud Boost"
09:29 "Verizon iPhone Faces Even Bigger Foe"
09:33 "Trujillo Says 'Many Questions' Remain For Verizon IPhone: Video"
09:45 "Verizon IPhone Event Live Blog"
09:45 "[video] News Hub: Verizon iPhone: How will AT&T Respond? [2.7 min]"
09:52 "Verizon's Big iPhone Unveil"
09:52 "[video] Verizon's iPhone Buzz"
09:54 "Mobile Broadband Subs Seen Hitting 1 Billion In 2011"
09:58 "[audio] Opening Bell from MarketWatch Radio Network [1.0 min]"
10:03 "Video: Is the Verizon iPhone a Threat to Android?"
10:05 "Verizon iPhone expected to have unlimited data plan"
10:14 "[video] iPhone or Not, Verizon Is A Winner"
10:19 "Verizon iPhone: Live Blog"
10:37 "Verizon: Stay Tuned"
10:47 "LIVE: The Verizon iPhone Event"
10:53 "Faber Report: What Will Drive Verizon?"
10:54 "LIVE FROM NEW YORK: Verizon Gets the iPhone"
10:54 "Verizon iPhone: Will It Fix Verizon's $100 Billion Conundrum?"
10:55 "Live Blog: Verizon's iPhone Announcement"
10:57 "Live Blogging the Verizon iPhone Announcement"
11:18 "Verizon to launch iPhone 4 in February"
11:20 "Verizon iPhone: The basics"
11:24 "The Biggest Surprise About the Verizon iPhone: It's a Mobile Hotspot"
11:24 "Verizon to Carry Apple iPhone, Ending Months of Anticipation"
11:30 "Verizon Will Be Selling The iPhone Starting February 10th"
11:36 "Live Blog: Verizon's iPhone Announcement"
11:38 "UPDATE 2-Verizon Wireless to sell iPhone in February"
11:39 "PHOTOS: Verizon iPhone Has Different (Better?) Antenna Design Than AT&T iPhone"
11:39 "Verizon, AT&T, And Apple: Guess Which One's Stock Isn't Tanking Today"
11:41 "Verizon to Sell iPhone 4 on Feb. 10"
11:43 "Verizon Wireless to Offer iPhone 4 in February"
11:45 "Verizon iPhone: 'It Begins'"
11:50 "Verizon to begin selling iPhone 4 in February"
11:51 "Bizarre Fact of the Day: Tyler Durden and Louis Winthorpe III Are Practically the Same Person"
11:51 "Verizon to sell Apple iPhone"
11:51 "Verizon: Began Discussions With Apple In 2008; Tim Cook Talks"
11:54 "Verizon Wireless to sell iPhone in February"
12:17 "AT&T Preps for Verizon iPhone With New Ads, Price Cuts"
12:18 "[audio] Enderle: Verizon and Apple no match made in heaven [5.1 min]"
12:19 "Apple's Phil Schiller on Building the Verizon iPhone"
12:20 "AT&T Shrugs Off Verizon iPhone"
12:20 "COMPARISON: Verizon Vs. AT&T iPhone"
12:26 "FACTBOX - AT&T and Verizon wage battle over iPhone"
12:28 "Verizon iPhone: Yep, The Real Target is Google."
12:28 "Why No 4G iPhone?"
12:30 "AT&T: Five Reasons the Sky Won't Fall When iPhone Goes to Verizon"
12:35 "[audio] Midday Update from MarketWatch Radio Network [1.0 min]"
12:37 "FACTBOX - AT&T and Verizon wage battle over iPhone"
12:37 "Verizon Wireless to sell iPhone in February"
12:43 "Verizon to start selling iPhone on Feb. 10"
12:48 "Corrected: Factbox: AT&T and Verizon wage battle over iPhone"
12:50 "Verizon Will Offer the iPhone Next Month"
12:57 "Corrections: Verizon to begin selling iPhone 4 in February"
12:59 "Verizon Wireless to sell iPhone in February"
13:17 "Stronger earnings reports push stocks higher"
13:18 "Most active New York Stock Exchange-traded stocks"
13:18 "[video] Apple's iPhone 4 Comes to Verizon [1.4 min]"
13:19 "iPhone Will Bring 'Value' to Verizon Shareholders: CEO"
13:22 "Apple/Verizon Pt. 2: Should I stay or should I go now?"
13:23 "FACTBOX: AT&T and Verizon wage battle over iPhone"
13:23 "Halftime: Verizon iPhone To Drag Down Verizon Stock 10%?"
13:25 "[$$] iPhone Disappears Over the Verizon"
13:28 "[audio] Time for a tech detox? [1.9 min]"
13:29 "Wireless Struggles Against FCC Over Possible ""Bill Shock"" Rules"
13:34 "Verizon big winner from having iPhone? Not so fast"
13:38 "Verizon President Talks iPhone Strategy"
13:43 "FACTBOX: AT&T and Verizon wage battle over iPhone"
13:43 "US stocks up on strong earnings [at Financial times]"
13:45 "Verizon Gets the iPhone"
13:45 "Verizon to start selling iPhone on Feb. 10"
13:49 "Apple: No LTE iPhone This Year?"
13:53 "AT&T: We're ""Evaluating"" The New iPhone Mobile Hotspot Feature"
13:58 "Apple: Surprise! Street Expresses Delight With VZ iPhone"
14:00 "[$$] Catching the Wave: Euro Rally Due"
14:02 "Did Apple disappoint with no 4G iPhone?"
14:03 "Mobile Payments: What Verizon's iPhone Means for Square"
14:04 "Verizon to Sell Apple IPad That Connects Directly to Its Network"
14:09 "Verizon Wireless ends long wait for iPhone fans"
14:10 "UPDATE 4-Verizon Wireless ends long wait for iPhone fans"
14:13 "Verizon Wireless ends long wait for iPhone fans"
14:25 "[audio] Before the Close from MarketWatch Radio Network [1.0 min]"
14:27 "Verizon Wireless ends long wait for iPhone fans"
14:27 "Verizon Wireless ends long wait for iPhone fans"
14:33 "Market Snapshot: U.S. stocks rise on earnings optimism"
14:33 "Verizon to start selling iPhone on Feb. 10"
14:41 "Analysts Go Out on Limb, Predict Verizon iPhone Will Be Big for Apple"
14:48 "Verizon Says You Can Switch If Santa Brought You a Droid"
14:49 "Verizon Wireless to Start IPhone Sales Next Month, Ending AT&T Exclusivity"
14:50 "Motorola: Opportunity In Lack Of LTE iPhone"
14:50 "Verizon iPhone, WoW: Hot Trends"
15:07 "Is Apple's iPhone a Scrooge?"
15:09 "How does the Verizon iPhone affect Android and AT&T?"
15:09 "More talk of smartphones diving into featurephone ground"
15:09 "Verizon iPhone unlimited data won't last forever"
15:09 "Why the Verizon 'dream phone' matters"
15:22 "[video] digits: Verizon Unveils Its iPhone [8.7 min]"
15:35 "With Verizon, Will Apple Take a Bite of More Enterprise?"
15:38 "5 'Dogs of the Dow' Worth Betting On"
15:38 "[video] iPhone: No wonders for Verizon stock"
15:44 "Why RadioShack Is Still Relevant"
15:50 "Apple: Jefferies Ups Tgt To $450; LTE iPhone In October?"
15:50 "BlackBerry PlayBook vs. Android Tablets"
15:51 "Happy Verizon iPhone Day!"
15:57 "[$$] Overheard"
15:58 "Verizon to start selling iPhone on Feb. 10"
16:12 "Verizon's Daniel Mead Doesn't Expect IPhone Price War: Video"
16:17 "Verizon to start selling iPhone on Feb. 10"
16:18 "Android to Lose From Verizon iPhone?"
16:20 "Investing for Income in 2011 with Stocks and Options Part 3: High-Yield Stocks"
16:24 "Tech Stocks: Techs post modest gains, but AMD sinks"
16:26 "[audio] Closing Bell from MarketWatch Radio Network [1.0 min]"
16:30 "Apple: Reasons For An LTE iPhone This Year"
16:30 "[video] digits: Verizon iPhone: What You Need to Know [4.1 min]"
16:30 "[video] digits:What Verizon iPhone Means for Apple, Google [2.1 min]"
16:31 "US STOCKS-Energy shares lift Wall St in light volume"
16:32 "Don't Buy The Verizon iPhone Yet, Unless You NEED A New Phone ASAP"
16:48 "[video] PM Report: China Stealth Fighter - Threat to U.S.? [10.2 min]"
16:50 "Stronger earnings reports push stocks higher"
17:50 "[video] Options Action"
17:54 "Stronger earnings reports push stocks higher"
17:55 "Factbox: AT&T and Verizon wage battle over iPhone"
18:09 "All Our iPhone Trades, Right Here & Now"
18:09 "Most active New York Stock Exchange-traded stocks"
18:44 "Doherty Sees 'Exodus' of AT&T IPhone Users to Verizon: Video"
19:00 "Lady Gaga Brags, U2 to Rival R.E.M. for 2011 Rock Crown: Preview"
19:40 "'Fast Money' Recap: Commodities Getting Frothy"
20:21 "MiFi Come, MiFi Go: Verizon Wireless To Sell iPads With 3G Connectivity"
22:02 "Tuesday ETF Roundup: UNG Surges on Weather, IYZ Tumbles on Increased Competition"
23:38 "Adding Up Parts at Goldman Sachs"

AT&T (T): Tuesday January 11th, 2011

00:01 "IPhone May Cost Verizon Wireless $5 Billion in Subsidies in First Year"
00:15 "ANALYSIS - AT&T faces tough year after losing iPhone exclusive"
02:47 "Telecoms' Dividend Yields"
03:03 "With iPhone Taunts, AT&T-Verizon Rivalry Escalates"
05:00 "[video] Verizon iPhone Data Plan"
05:57 "Cramer's Lightning Round - Viva Las Vegas (1/10/11)"
06:00 "Nielsen: Young People Across The Globe Love Their Cell Phones (But Use Them Differently)"
06:21 "[$$] RadioShack Needs Verizon Wireless"
06:45 "Half of Dow Companies to Post Profit Gains"
06:50 "Ahead of the Bell: Verizon iPhone"
07:15 "GM returning to Super Bowl for first time since '08."
07:27 "10 Things You Need To Know Before The Opening Bell"
07:27 "10 Things You Need To Know Before The Opening Bell"
07:32 "Lowenstein Says Verizon's Network Ready to Handle IPhone: Video"
07:40 "iPhone Raises Stakes of AT&T-Verizon Rivalry"
07:41 "10 Things You Need to Know Before the Opening Bell"
07:45 "Verizon big winner from having iPhone? Not so fast"
07:58 "AT&T and Verizon Trade Taunts Over iPhone"
08:03 "[video] 3 Stocks I Saw on TV"
08:05 "AT&T-Verizon rivalry kicks up a notch"
08:09 "10 Unusual Stocks Attracting Huge Interest This Morning"
08:22 "Will Verizon Demolish AT&T? It Did So Long Ago"
08:24 "The Big Questions About Verizon's IPhone"
08:24 "Verizon big winner from having iPhone? Not so fast"
08:25 "McCourt Says Verizon's Network Can Handle Apple's IPhone: Video"
08:36 "How Important Is Offering the iPhone on Verizon to Apple?"
08:45 "[$$] ETF Play of the Day: XLK"
09:02 "[video] AM Report: Chinese Stealth Jet Takes Flight [8.0 min]"
09:06 "Three Things to Watch For at Verizon iPhoneapalooza"
09:29 "Verizon iPhone Faces Even Bigger Foe"
09:33 "Trujillo Says 'Many Questions' Remain For Verizon IPhone: Video"
09:45 "Verizon IPhone Event Live Blog"
09:45 "[video] News Hub: Verizon iPhone: How will AT&T Respond? [2.7 min]"
09:52 "[video] Verizon's iPhone Buzz"
09:54 "Mobile Broadband Subs Seen Hitting 1 Billion In 2011"
09:58 "[audio] Opening Bell from MarketWatch Radio Network [1.0 min]"
10:03 "Video: Is the Verizon iPhone a Threat to Android?"
10:05 "Verizon iPhone expected to have unlimited data plan"
10:14 "[video] iPhone or Not, Verizon Is A Winner"
10:19 "Verizon iPhone: Live Blog"
10:37 "Verizon: Stay Tuned"
10:43 "Mobile Broadband Subscriptions to Surpass 1 Billion in 2011"
10:47 "LIVE: The Verizon iPhone Event"
10:54 "LIVE FROM NEW YORK: Verizon Gets the iPhone"
10:54 "Verizon iPhone: Will It Fix Verizon's $100 Billion Conundrum?"
11:10 "Verizon to Partner With Apple, Start Selling IPhone Next Month"
11:14 "Verizon and Apple Make It Official"
11:15 "Verizon Wireless to Start IPhone Sales Next Month, Ending AT&T Exclusivity"
11:15 "iPhone Breaks AT&T Marriage Today for Verizon"
11:18 "Verizon to launch iPhone 4 in February"
11:20 "Verizon iPhone: The basics"
11:24 "The Biggest Surprise About the Verizon iPhone: It's a Mobile Hotspot"
11:24 "Verizon to Carry Apple iPhone, Ending Months of Anticipation"
11:30 "Verizon Will Be Selling The iPhone Starting February 10th"
11:38 "UPDATE 2-Verizon Wireless to sell iPhone in February"
11:39 "PHOTOS: Verizon iPhone Has Different (Better?) Antenna Design Than AT&T iPhone"
11:39 "Verizon, AT&T, And Apple: Guess Which One's Stock Isn't Tanking Today"
11:43 "Verizon Wireless to Offer iPhone 4 in February"
11:45 "Verizon iPhone: 'It Begins'"
11:50 "Verizon to begin selling iPhone 4 in February"
11:51 "Stocks Led by Energy, Conglomerates"
11:51 "Verizon to sell Apple iPhone"
11:51 "Verizon: Began Discussions With Apple In 2008; Tim Cook Talks"
11:54 "Verizon Wireless to sell iPhone in February"
12:08 "Verizon iPhone Will Not Work Overseas"
12:13 "UPDATE 3-Verizon Wireless to sell iPhone in February"
12:16 "FACTBOX - AT&T and Verizon wage battle over iPhone"
12:17 "AT&T Preps for Verizon iPhone With New Ads, Price Cuts"
12:19 "Apple's Phil Schiller on Building the Verizon iPhone"
12:20 "AT&T Shrugs Off Verizon iPhone"
12:20 "COMPARISON: Verizon Vs. AT&T iPhone"
12:26 "FACTBOX - AT&T and Verizon wage battle over iPhone"
12:28 "Verizon iPhone: Yep, The Real Target is Google."
12:30 "AT&T: Five Reasons the Sky Won't Fall When iPhone Goes to Verizon"
12:35 "[audio] Midday Update from MarketWatch Radio Network [1.0 min]"
12:37 "FACTBOX - AT&T and Verizon wage battle over iPhone"
12:37 "Verizon Wireless to sell iPhone in February"
12:43 "Verizon to start selling iPhone on Feb. 10"
12:48 "Corrected: Factbox: AT&T and Verizon wage battle over iPhone"
12:59 "Verizon Wireless to sell iPhone in February"
13:05 "Verizon iPhone set for Feb. 10 release"
13:05 "Verizon to start selling iPhone Feb. 3"
13:07 "Moore Says 3 Million AT&T IPhone Users May Go to Verizon: Video"
13:11 "Cook Says Apple-Verizon Cooperation 'Just the Beginning': Video"
13:12 "COMPARISON: Verizon Vs. AT&T iPhone"
13:18 "Most active New York Stock Exchange-traded stocks"
13:18 "Summary Box: Verizon to start selling iPhone"
13:18 "[video] Apple's iPhone 4 Comes to Verizon [1.4 min]"
13:22 "Apple/Verizon Pt. 2: Should I stay or should I go now?"
13:23 "FACTBOX: AT&T and Verizon wage battle over iPhone"
13:23 "Halftime: Verizon iPhone To Drag Down Verizon Stock 10%?"
13:29 "Wireless Struggles Against FCC Over Possible ""Bill Shock"" Rules"
13:34 "Verizon big winner from having iPhone? Not so fast"
13:43 "FACTBOX: AT&T and Verizon wage battle over iPhone"
13:43 "US stocks up on strong earnings [at Financial times]"
13:45 "Verizon to start selling iPhone on Feb. 10"
13:49 "Apple: No LTE iPhone This Year?"
13:53 "AT&T: We're ""Evaluating"" The New iPhone Mobile Hotspot Feature"
13:58 "Apple: Surprise! Street Expresses Delight With VZ iPhone"
14:02 "Did Apple disappoint with no 4G iPhone?"
14:04 "Verizon to Sell Apple IPad That Connects Directly to Its Network"
14:05 "Verizon begins selling iPhone on Feb. 3"
14:05 "Verizon iPhone sales begin Feb. 3"
14:09 "Verizon Wireless ends long wait for iPhone fans"
14:10 "UPDATE 4-Verizon Wireless ends long wait for iPhone fans"
14:13 "Verizon Wireless ends long wait for iPhone fans"
14:25 "[audio] Before the Close from MarketWatch Radio Network [1.0 min]"
14:27 "Verizon Wireless ends long wait for iPhone fans"
14:27 "Verizon Wireless ends long wait for iPhone fans"
14:33 "Market Snapshot: U.S. stocks rise on earnings optimism"
14:33 "Verizon to start selling iPhone on Feb. 10"
14:41 "Analysts Go Out on Limb, Predict Verizon iPhone Will Be Big for Apple"
14:50 "Motorola: Opportunity In Lack Of LTE iPhone"
14:57 "An iPhone For Valentine's Day and iPad 2 Update"
14:58 Verizon
15:05 "Verizon iPhone sales begin Feb. 3"
15:05 "Verizon iPhone set for Feb. 10 release"
15:09 "How does the Verizon iPhone affect Android and AT&T?"
15:09 "More talk of smartphones diving into featurephone ground"
15:09 "Verizon iPhone unlimited data won't last forever"
15:09 "Why the Verizon 'dream phone' matters"
15:38 "5 'Dogs of the Dow' Worth Betting On"
15:38 "[video] iPhone: No wonders for Verizon stock"
15:50 "Apple: Jefferies Ups Tgt To $450; LTE iPhone In October?"
15:51 "Happy Verizon iPhone Day!"
15:58 "Verizon to start selling iPhone on Feb. 10"
16:06 "Verizon iPhone Is Not Mobile Messiah: Analyst"
16:10 "Stocks March Higher As Earnings Season Begins"
16:12 "Verizon's Daniel Mead Doesn't Expect IPhone Price War: Video"
16:17 "Verizon to start selling iPhone on Feb. 10"
16:18 "Android to Lose From Verizon iPhone?"
16:20 "Investing for Income in 2011 with Stocks and Options Part 3: High-Yield Stocks"
16:28 "Daley Has $7.7 Million of JPMorgan Stock to Divest on Way to White House"
16:30 "Apple: Reasons For An LTE iPhone This Year"
16:32 "Don't Buy The Verizon iPhone Yet, Unless You NEED A New Phone ASAP"
16:50 "Stronger earnings reports push stocks higher"
17:04 "Verizon Wireless ends long wait for iPhone fans"
17:08 "Verizon Wireless ends long wait for iPhone fans"
17:22 "Verizon Wireless ends long wait for iPhone fans"
17:22 "Verizon Wireless ends long wait for iPhone fans"
17:41 "California Reclaims 48,000 State-Owned Mobile Phones"
17:43 "[video] News Hub: Verizon iPhone - What's The Actual Cost? [3.0 min]"
17:54 "Stronger earnings reports push stocks higher"
17:55 "Factbox: AT&T and Verizon wage battle over iPhone"
18:09 "All Our iPhone Trades, Right Here & Now"
18:09 "Most active New York Stock Exchange-traded stocks"
18:44 "Doherty Sees 'Exodus' of AT&T IPhone Users to Verizon: Video"
20:21 "MiFi Come, MiFi Go: Verizon Wireless To Sell iPads With 3G Connectivity"
22:02 "Tuesday ETF Roundup: UNG Surges on Weather, IYZ Tumbles on Increased Competition"
23:38 "Adding Up Parts at Goldman Sachs"

Merck (MRK): Thursday January 13th, 2011

08:58 "JP Morgan Healthcare: Sanofi-Aventis, UCSF in Research Pact"
09:20 "Merck Statement on Changes to Clinical Studies of Vorapaxar"
09:41 "U.S. stocks open lower as joblessness rises"
09:41 "UPDATE 1-Merck stops giving clot drug in study, shares drop"
09:47 "Merck Drags on Stocks"
09:49 "Merck tumbles on drug study woes"
09:55 "Market Snapshot: U.S. stocks open lower as joblessness rises"
09:58 "Merck Blood-Thinner Study Ends in Stroke Patients, Limiting Possible Sales"
09:58 "Stocks Fall Slightly After Jobless Claims, PPI"
10:56 "Movers & Shakers: Thursday?s biggest gaining and declining stocks"
11:06 "Idaho in Drug Settlement With Warrick, Schering-Plough"
11:10 "Stocks Weighed by Healthcare Sector"
11:19 "[video] Markets Hub: US Stocks Fall As Merck Leads Decline [2.9 min]"
11:28 "FDA seeks less acetaminophen in prescription drugs"
11:34 "Commodities Look 'Very Attractive' Now: Kass"
11:41 "The Cheaper Way to Build the Best Portfolio"
11:43 "Biotech Stocks: Merck leads drug stocks lower"
11:43 "UPDATE 3-Merck clot drug seen unfit for stroke, shares fall"
11:47 "Merck Pipeline Setback: Big Pharma Losers"
11:49 "Merck clot drug seen unfit for stroke, shares fall"
11:53 "US STOCKS-Market flat, pares losses from jobless claims"
11:54 "Markets Going Nowhere After Weak US Data: Here Are The 10 Trades To Watch"
11:56 "Merck Learns There Is No Magic Pill"
11:56 "U.S. stocks bounce off lows; oil turns up"
12:16 "Bond Report: Treasurys rise after Fed purchases"
12:27 "Merck Shares Drop After Anti-Clotting Drug Trials Curtailed"
12:39 "Market Snapshot: U.S. stocks mixed as joblessness rises"
12:45 "Market flat, pares losses from jobless claims"
12:47 "A Painful Day for Merck"
13:05 "Merck tumbles on drug study update"
13:10 "US STOCKS-Market flat on weak jobless claims, Marathon's rise"
13:12 "Market flat, pares losses from jobless claims"
13:20 "Jobless claims weigh on US equities"
13:24 "Market flat on weak jobless claims, Marathon's rise"
13:29 "Dealpolitik: The Useful Corruption of Shareholder Lawsuits"
13:44 "Does Drug Setback Call Merck's Schering-Plough Purchase Into Question?"
13:47 "UPDATE 4-Merck clot drug seen unfit for stroke, shares fall"
13:48 "Market flat on weak jobless claims, Marathon's rise"
14:27 "Market Snapshot: Merck leads stocks lower"
14:35 "Merck Hurts, Stocks Stagger After Jobless Claims"
14:45 "Drug Study Torpedos Merck Stock"
14:52 "Bond Report: Treasurys lifted by Fed buys; 30-year sale in line"
14:55 "Jan. 13: Unusual Volume Leaders"
15:00 "Credit Suisse: All Isn't Lost for Merck"
15:12 "Merck weighs on Dow, market eyes earnings"
15:23 "This Trendy Pharma Is Making a Stylish Move"
15:36 "The Three Most Profitable Healthcare Trends of 2011"
15:47 "Market Snapshot: Stocks slide into close"
15:55 "Copying biotech medicine attracts more drugmakers"
16:00 "Stocks Slip Up On Awful US Data: Here's What You Need To Know"
16:08 "US STOCKS SNAPSHOT-Wall St dips on Merck, commodities"
16:09 "The Big Culprit In Today's DOW Drop"
16:09 "UPDATE 1-FDA seeks less acetaminophen in prescription drugs"
16:10 "U.S. stocks end lower as joblessness rises"
16:27 "US STOCKS-Merck, materials companies drag Wall St lower"
16:27 "[audio] Closing Bell from MarketWatch Radio Network [1.0 min]"
16:55 "Bond Report: Treasurys rise with jobless claims, auctions? end"
16:56 "[video] PM Report: Economists Turn Upbeat on Recovery [10.6 min]"
16:59 "[$$] Stocks in the Spotlight Thursday"
17:06 "US STOCKS-Merck, materials drag Wall St lower, Intel up late"
17:09 "[video] News Hub: Dow Dragged Down by Merck [1.3 min]"
17:11 "Market Snapshot: Stocks end lower on weak materials, drug shares"
18:29 "The Fed Takes Credit: Dave's Daily"
18:34 "[audio] After the Bell from MarketWatch Radio Network [1.0 min]"
18:39 "Trouble With Thrombin Inhibitor Takes a Toll on Merck Shares"
18:59 "Thursday ETF Roundup: UNG Sinks on Natural Gas Data, BLV Rises on Bond Boost"
20:06 "[$$] Options Traders Target Merck After Drug Study Halted"
20:26 "HK stocks seen lower on weak materials sector"

Bank of America (BAC): Friday January 14th, 2011

00:01 "Google, Renault, RBS, Stanford, Kissel, Madoff in Court News"
00:01 "Lehman, Madoff, FGIC, Chemtura, FairPoint, Quigley: Bankruptcy"
00:02 "Geithner starts talks on tax overhaul"
01:07 "Morgan Stanley's Gorman shakes up management"
01:25 "Morgan Stanley's Gorman shakes up management"
02:06 "Banks Are Poised to Pay Dividends After 3-Year Gap"
03:12 "UPDATE 1-India govt to decide on IOC share sale in 3-4 mths"
04:01 "What's Driving Earnings Higher?"
06:00 "5 Bank Stocks Under $5"
06:00 "5 Land Mines in Fourth Quarter Bank Earnings"
06:00 "[video] Hold, Don't Buy Bank Stocks: Fund Manager"
06:59 "Kissel Was Argumentative, Easily Upset, Sister-In-Law Testifies"
07:00 "Filling In FINRA's Gaps"
07:00 "General Electric Earnings: What to Expect"
07:00 "Worst-Performing Financial CEO of 2010"
07:16 "[$$] Earnings Scorecard: Commercial and Investment Banks"
07:18 "JPMorgan profit surges despite mortgage hit"
07:36 "JPMorgan's Fourth Quarter Profit Rises 47% to $4.8 Billion"
07:48 "RPT-UPDATE 4-Morgan Stanley's Gorman shakes up management"
07:59 "J.P. Morgan's Fourth-Quarter Profit Jumps 47%"
08:04 "Floods take major toll on Australian farms, mines"
08:08 "A Stock-Bond Hybrid That Flopped"
08:08 "Big Banks with Big Legal Bills"
08:08 "Home Depot's Fix-It Lady"
08:08 "Why the Private Sector Still Isn't Hiring"
08:12 "Citigroup, Bank of America Active Premarket After JPM Report"
08:19 "What's On: JPMorgan, Intel and Corporate Taxes"
08:24 "UPDATE 4-JPMorgan profit beats, helped by reserve release"
08:26 "Hybrid Debt, Materials Curbs, Tax Cheats: Compliance"
08:26 "JPMorgan profit beats, helped by reserve release"
08:37 "Deals of the Day: Groupon Talks IPO"
08:53 "GLOBAL MARKETS WEEKAHEAD-Reality checks ahead for investors"
08:54 "J.P. Morgan Earnings: Were They As Good as They Looked?"
08:55 "J.P. Morgan Earnings: Still Building Litigation Warchest"
09:01 "[$$] ETF Play of the Day: KBE"
09:10 "Foreclosure Wave Cresting, May Never Break"
09:40 "Earnings Jump At JPMorgan, But Street Stifled By China"
09:42 "U.S. Probes Banks' Mortgage Practices"
09:46 "JPMorgan profit rises 47 percent"
09:49 "J.P. Morgan: Mortgage Fights a 'Long, Ugly Mess'"
09:52 "ManTech International - Upgrades & Downgrades"
09:56 "Lbma Says Merrill Lynch Reclassified as Options Market Maker"
09:57 "UBS Upgrades Financials"
10:04 "American Capital prices share offer at $28 each"
10:09 "Schwab Takes High-Profile Stand Opposing Self-Regulatory Organization"
10:12 "BofA Suffers Online Glitches"
10:16 "JPMorgan: Strong Earnings, Muted Response"
10:23 "Stocks edge up after tepid inflation report"
10:33 "JPM Dividend: Jamie Dimon Balks"
10:34 "Your Index Fund Is Making a Bad Investment"
10:43 "Apple's Magic Share Price: $422"
10:45 "JPMorgan Chase Tops 4Q Estimates. Citigroup Is Up Next."
10:58 "Citigroup: Uncle Sam to Get Out, Completely"
11:01 "[$$] JPMorgan Chase Delivers Insight"
11:05 "Affiliated Managers gets $750M credit line"
11:08 "A Wave of Mergers And Acquisitions for Northeast And Mid-Atlantic Banks: An Exclusive Interview With Stephen M. Moss Of Janney Montgomery Scott LLC"
11:11 "UPDATE 6-JPMorgan profit rises 47 pct, beating estimates"
11:11 "[$$] Big Banks May Boost Dividends in 2011"
11:13 "New Zealand bourse winds up AXE trading venue"
11:20 "Greenspan, Fed Saw Threat of Stagflation After Katrina Devastation in 2005"
11:25 "A Wall Street Mystery: Where Was Goldman's CEO?"
11:29 "Goldman Lost an Extra $5B in Prop Trading: Report"
11:29 "JPMorgan profit rises 47 percent"
11:49 "[video] How JPMorgan's Quarter Affects Bank Stocks"
11:53 "Banks lead stocks higher, led by JPMorgan Chase"
12:03 "JPMorgan Earnings Lift Bank Stocks"
12:03 "John Paulson's Top Picks"
12:04 "CPI Shows Input-Cost Inflation Tempered By Overcapacity"
12:05 "Bank of America website having problems"
12:05 "JPMorgan Chase profit up 47%"
12:08 "Earnings schedule for week of 1/17/2011"
12:10 "CNNMoney stock market report -- Jan. 14, 2011"
12:26 "Banks lead stocks higher, led by JPMorgan Chase"
12:29 "J.P. Morgan?s profit jumps 47%"
12:36 "[$$] ETFs for the Dividend-Minded Investor"
13:44 "UPDATE 7-JPMorgan beats, sets bullish tone for bank earnings"
13:48 "AIG recap deal closes, focus moves to share sale"
13:48 "JPMorgan sets bullish tone for bank earnings"
13:50 "[video] J.P. Morgan Earnings Rise 47%"
13:52 "New Movie Revisits the Last 24 Hours of Lehman Brothers, or a Very Similar Firm"
13:52 "[$$] J.P. Morgan Gives Dow Modest Boost"
13:54 "Fed?s Tarullo Tells CNBC No Reason to Adjust Asset Purchases"
13:55 "Should You Buy 2010's Best Investments?"
14:00 "Banking On A Better 2011"
14:02 "[video] Rush Card"
14:05 "JPMorgan Credit Swaps Climb as Bank Results Rest on Reserves"
14:06 "How Celebrities Buy And Sell Homes"
14:07 "A female quota at Davos? Really?"
14:26 "JPMorgan Rises On Double Beat"
14:40 "Bank of America, Wells Fargo May Lose Under Plan to Simplify Mortgages"
14:43 "PNC, Capital One Advance After JPMorgan Posts Record Profit"
14:45 "Bank of America online banking down for some users"
14:45 "Banks take stock indexes higher"
14:56 "Financial Stocks: Financial shares higher after J.P. Morgan results"
15:01 "Halftime: Trading Financials Ahead of Earnings Next Week"
15:05 "Bank of America website having problems"
15:10 "Bank of America website suffers outage"
15:16 "AIG recap deal closes, focus moves to share sale"
15:21 "JPMorgan beats, sets bullish tone for bank earnings"
15:27 "PREVIEW-UPDATE 1-Volcker rule tests new US systemic risk council"
15:29 "Bank of America online banking down for some users"
15:37 "Bank of America Downplays Web Problems"
15:55 "Volcker rule tests new US systemic risk council"
15:57 "AIG recap deal closes, focus moves to share sale"
15:57 "Bank of America Website is Down For Some"
15:59 "[$$] Treasurys Fall, Giving Up Earlier Rally; Little Change for Week"
16:10 "Assured Guaranty was only muni bond insurer in 2010"
16:17 "U.S. stocks post seventh week of gains"
16:18 "Great Depression Part Deux Averted But Bubble Machine Still Blows"
16:18 "Volcker rule tests new US systemic risk council"
16:23 "Tax talks get good start, companies say"
16:29 "Banks take stock indexes higher, led by JPMorgan"
16:31 "Dow, S&P 500 Extend Seven-Week Winning Streak"
16:31 "[$$] Interesting Rotation in the Big Banks"
16:36 "Banks lead S&P 500 to seventh week of gains"
16:52 "Stocks Reach New Records"
16:54 "Market Snapshot: U.S. stocks rise to 30-month highs"
16:55 "Trading Radar: Key Earnings, China Headline the Week"
17:41 "JPMorgan beats, sets bullish tone for bank earnings"
18:15 "AIG recapitalization deal closes, share sale looms"
18:32 "Wall St Week Ahead: Investors crave more strong bank results"
18:46 "GLOBAL MARKETS WEEKAHEAD - Reality checks ahead for investors"
18:46 "US STOCKS - Banks lead S&P 500 to seventh week of gains"
18:46 "Volcker rule tests new US systemic risk council"
20:55 "AIG recapitalization deal closes, share sale looms"
20:55 "JPMorgan beats, sets bullish tone for bank earnings"

JP Morgan (JPM): Friday January 14th, 2011

07:29 "JPMorgan Profit Beats on Narrowing Loan Losses"
07:29 "JPMorgan reports jump in quarterly profit"
07:33 "US STOCKS-Futures fall, JPMorgan profit jumps"
07:34 "JPMorgan Net Rises 47% on Lower Credit Costs, Tops Estimates"
07:35 "Futures Lower Despite Earnings Beats"
07:35 "[audio] Before the bell from MarketWatch Radio Network [1.0 min]"
07:36 "JPMorgan's Fourth Quarter Profit Rises 47% to $4.8 Billion"
07:37 "J.P. Morgan posts quarterly profit of $4.8 billion"
07:39 "UPDATE 3-JPMorgan profit beats, helped by reserve release"
07:40 "The Good Leads"
07:41 "JPMorgan Chase's profit jumps 47 percent in 4Q"
07:41 "JPMorgan quarterly profits surge 47%"
07:42 "JPMorgan profit beats expectations"
07:42 "UPDATE 1-TABLE-JPMorgan fourth-quarter results"
07:45 "CNNMoney premarket report -- Jan. 14, 2011"
07:48 "RPT-UPDATE 4-Morgan Stanley's Gorman shakes up management"
07:56 "[$$] J.P. Morgan Profit Jumps 47%"
07:57 "Indications: U.S. stock futures point to soft open"
07:57 "Stock futures off, Dow, S&P seek 7th week of gains"
07:59 "J.P. Morgan's Fourth-Quarter Profit Jumps 47%"
08:26 "JPMorgan profit beats, helped by reserve release"
08:28 "JPMorgan Beats Street, Net Rises 47%"
08:28 "Morning Take-Out"
08:30 "Minyanville's T3 Morning Call: JPM, INTC Earnings Beats Can't Lift Futures"
08:30 "[$$] The Day Ahead: Follow the Leaders"
08:31 "JP Morgan results ahead of forecasts"
08:36 "[$$] It's All Baked In"
08:41 "Deals wrap: JPMorgan surprises"
08:41 "[$$] Futures Stay Weak After U.S. Data"
08:43 "M&T Beats Estimates"
08:44 "US STOCKS-Futures briefly tick lower after CPI, retail sales"
08:45 "Futures Hold Losses After CPI, Retail Sales"
08:46 "[$$] Will Intel Be the Downside Catalyst?"
08:52 "Live coverage of JPMorgan?s earnings call"
08:54 "J.P. Morgan Earnings: Were They As Good as They Looked?"
08:55 "J.P. Morgan Earnings: Still Building Litigation Warchest"
08:56 "JPMorgan Chase's profit jumps 47 percent in 4Q"
08:57 "JPMorgan reports jump in quarterly profit"
09:38 "CNNMoney stock market report -- Jan. 14, 2011"
09:38 "JPMorgan's Biggest Cost? Pay, Not Mortgages"
09:38 "UPDATE 5-JPMorgan profit rises 47 pct, beating estimates"
09:40 "Earnings Jump At JPMorgan, But Street Stifled By China"
09:41 "Financials open higher after J.P. Morgan earnings"
09:41 "Stocks Slip at Open After Mixed Economic News"
09:42 "U.S. Probes Banks' Mortgage Practices"
09:45 "Unique Convert Offering From Dendreon via JPMorgan"
09:46 "JPMorgan profit rises 47 percent"
09:47 "Friday Look Ahead: Intel, JPMorgan Earnings and Consumer Mood"
09:48 "J.P. Morgan?s profit jumps 47%"
09:49 "J.P. Morgan: Mortgage Fights a 'Long, Ugly Mess'"
09:50 "Stock Market Story: Jan. 14"
09:57 "JPMorgan Reports Q4 Results"
09:57 "Markets dip after JPMorgan earns, retail sales"
09:57 "UBS Upgrades Financials"
10:16 "JPMorgan: Strong Earnings, Muted Response"
10:23 "Stocks edge up after tepid inflation report"
10:24 "[$$] JPMorgan Worries Down the Drain"
10:25 "[$$] Mixed Start"
10:28 "US open: Dow trades flat"
10:30 "[$$] A Short-Term Retail Play"
10:31 "China's inflation move weighs on Wall St"
10:33 "JPM Dividend: Jamie Dimon Balks"
10:45 "[$$] Dip-Buyers' Delight"
10:47 "UPDATE 1-JPMorgan Q4 commods risk steady after market rally"
10:54 "Friday's ETF to Watch: Merrill Lynch Regional Bank HOLDR"
10:55 "J.P. Morgan banker pay slips -- to $369,651"
10:57 "Markets flat as JPMorgan results offset Intel"
10:58 "Citigroup: Uncle Sam to Get Out, Completely"
10:58 "Financial Stocks: Financial shares higher after J.P. Morgan results"
11:08 "A Wave of Mergers And Acquisitions for Northeast And Mid-Atlantic Banks: An Exclusive Interview With Stephen M. Moss Of Janney Montgomery Scott LLC"
11:11 "[$$] Big Banks May Boost Dividends in 2011"
11:16 "[$$] J.P. Morgan Bankers Earn Less"
11:20 "Greenspan, Fed Saw Threat of Stagflation After Katrina Devastation in 2005"
11:20 "JPMorgan Paces Bank Gains After Earnings"
11:25 "A Wall Street Mystery: Where Was Goldman's CEO?"
11:26 "Debit fee caps may hurt poorest customers: Dimon"
11:28 "Market Snapshot: Stocks edge up as J.P. Morgan lifts banks"
11:29 "Goldman Lost an Extra $5B in Prop Trading: Report"
11:29 "JPMorgan profit rises 47 percent"
11:29 "News Corp. in Talks to Buy Shine"
11:33 "JPM Earnings"
11:41 "Complacency, Fleeing Insiders Point to Earnings Season Drop"
11:42 "Gold Prices Tank on Moves to Curb Inflation"
11:42 "JPMorgan: Bonus Numbers Look Good But Bankers Are Wary"
11:46 "UPDATE 2-TABLE-JPMorgan fourth-quarter results"
11:49 "[video] How JPMorgan's Quarter Affects Bank Stocks"
11:53 "Banks lead stocks higher, led by JPMorgan Chase"
12:03 "JPMorgan Earnings Lift Bank Stocks"
12:05 "Bank of America website having problems"
12:05 "JPMorgan Chase profit soars 47%"
12:05 "JPMorgan Chase profit up 47%"
12:09 "A Buying Opportunity Before Stress-Test Round 2?"
12:10 "S&P 500 Is Poised to Drop; RBC, JPMorgan Differ on the Timing"
12:11 "[$$] Comeback After Dazzling Comeback"
12:12 "US STOCKS-Market rises as JPMorgan profit outweighs data"
12:26 "Banks lead stocks higher, led by JPMorgan Chase"
12:32 "[audio] Midday Update from MarketWatch Radio Network [1.0 min]"
12:39 "Hey, Jamie Dimon, You Sure About that Acronym?"
12:55 "Stocks hurt by global inflation concerns"
12:56 "China, U.S. Weigh on Europe's Markets"
13:10 "Market Snapshot: Stocks find footing J.P. Morgan lifts banks"
13:11 "US STOCKS-Wall St rises as JPMorgan earnings spur optimism"
13:14 "Spain, Portugal And A Virtuous Circle Of Bailouts"
13:20 "Most active New York Stock Exchange-traded stocks"
13:34 "4 Top Financials for 1Q from Bernstein Research"
13:36 "Bank Of America Website Down, 6 Hours And Counting"
13:43 "UPDATE 3-AIG recap deal closes, focus moves to share sale"
13:44 "JPMorgan Chase's income jumps 47 percent"
13:48 "AIG recap deal closes, focus moves to share sale"
13:48 "JPMorgan sets bullish tone for bank earnings"
13:50 "[video] J.P. Morgan Earnings Rise 47%"
13:52 "New Movie Revisits the Last 24 Hours of Lehman Brothers, or a Very Similar Firm"
13:53 "JPMorgan: JV Approval in China Positions Stock for Upside"
13:54 "Fed?s Tarullo Tells CNBC No Reason to Adjust Asset Purchases"
13:59 "Elisabeth Murdoch's Shine hires bank for review"
14:00 "Banking On A Better 2011"
14:03 "J P MORGAN CHASE & CO Files SEC form 8-K, Regulation FD Disclosure"
14:05 "JPMorgan Credit Swaps Climb as Bank Results Rest on Reserves"
14:06 "How Celebrities Buy And Sell Homes"
14:07 "A female quota at Davos? Really?"
14:10 "M&T Bank Tops Zacks Estimate"
14:17 "Summary Box: JPMorgan Chase's profit jumps in 4Q"
14:17 "What's Behind JP Morgan's $1.12 Shares"
14:20 "Hedge funds appeal ruling on WaMu trust securities"
14:26 "JPMorgan Rises On Double Beat"
14:31 "UK unmatched for bank bonus-bashing"
14:34 "VIX Falls to Three-Week Low After JPMorgan Posts Record Profit"
14:35 "US STOCKS-JPMorgan lifts Wall St, but banks due for pause"
14:40 "Bank of America, Wells Fargo May Lose Under Plan to Simplify Mortgages"
14:43 "PNC, Capital One Advance After JPMorgan Posts Record Profit"
14:45 "Banks take stock indexes higher"
14:50 "Explaining Confirmation"
14:58 "JPMorgan Chase Plans $3.25 Billion Debt Offering"
15:01 "Gold Prices Sink on Rising Inflation Concerns"
15:05 "Bank of America website having problems"
15:16 "AIG recap deal closes, focus moves to share sale"
15:17 "Bond Report: Treasurys fall as relief rally dissipates"
15:21 "JPMorgan beats, sets bullish tone for bank earnings"
15:22 "Market Snapshot: U.S. stocks rise as J.P. Morgan lifts banks"
15:27 "PREVIEW-UPDATE 1-Volcker rule tests new US systemic risk council"
15:39 "[$$] Crude Settles Above $91"
15:40 "Why Citi Won't Repeat JPMorgan's Success"
15:43 "What to Do With JP Morgan Now?"
15:50 "JPMorgan Chase Declares Preferred Stock Dividend"
16:15 "PNC, Capital One: Winners & Losers"
16:17 "U.S. stocks post seventh week of gains"
16:18 "Great Depression Part Deux Averted But Bubble Machine Still Blows"
16:18 "Volcker rule tests new US systemic risk council"
16:25 "[audio] Closing Bell from MarketWatch Radio Network [1.0 min]"
16:29 "Banks take stock indexes higher, led by JPMorgan"
16:30 "MarketWatch?s top 10 stories: Jan. 10-14"
16:31 "HCP, Kimco: REIT Broker Action"
16:31 "New Issue-JP Morgan sells $3.25 bln in 2 parts"
16:31 "US STOCKS-Banks lead S&P 500 to seventh week of gains"
16:31 "[$$] Interesting Rotation in the Big Banks"
16:32 "Coinstar, JPMorgan, Borders are big market movers"
16:48 "Stocks Buck Mixed Data to Head Higher"
16:50 "Buzz on the Street: Tech Stocks Continue to Lead the Way"
16:52 "Stocks Reach New Records"
16:54 "Market Snapshot: U.S. stocks rise to 30-month highs"
16:55 "[video] News Hub: Dow Ends at 2-1/2 Year High [1.5 min]"

Ceci n'est ce pas une Haskell Monad Tutorial

2010-12-13T19:00:00.008Z

With apologies to the great surrealist painter Magritte, this really isn't a Haskell Monad Tutorial (HMT), or is it?

If you are reading this, you are probably aware that the blogosphere is littered with HMTs. The progression usually goes like this:

A programmer...

undertakes to learn Haskell,
struggles for some period of time with understanding Monads,
suddenly has that "ah-ha" moment, and
writes a tutorial (convinced that they can finally explain it to the beginner better than anyone else who has gone before could).

In my case, I choose to diverge from this pattern (slightly):

I...

took it upon myself to learn Haskell
struggled with Monads for some time
kind of slowly and grudgingly eventually "got" monads, but not with any kind of dramatic "eureka!", more like a "oh... right... ok" kind of feeling (which was remarkably anti-climactic considering how much hair I'd pulled out over this), and
am writing not so much a tutorial as a: "Here's what I did to understand it. Hope it helps."

I am not going to talk about Monad laws, IO, or Maybe. I am not even going to try to concoct a flowery definition of what it is (like "contanier" or "computational context"). I am just going to give two examples, which I wrote, specificially with a view to clarifying my own understanding (as per the suggestion of Brian Beckman, see video link below).

If this helps you, great. I wrote this to help me, and it served that purpose. It's called "learn by doing" (practical, rather than theoretical). I therefore must also suggest that you try this yourself, rather than just expect to get it by reading this. I'm willing to concede that there are probably serious problems with these definitions, given that I am not a schooled Haskell programmer, and don't know a lot about the conventions that serious people follow.

I'd like to start by throwing down a few references which were helpful to me in my understanding:

Brian Beckman: Don't Fear the Monads (video)
Greg Meredith: Intro to Monads (video)
Learn You a Haskell for Great Good (best "book" for beginners)
James Iry's blog: One Div Zero

Now, those folks really know what they're talking about. Look to them for a real education. But, if you still want to see my rudimentary examples, here they are...

The two examples I implemented: the identity monad (the simplest of all!), and the state monad (because I really wanted to understand this for a project that I'm working on). Implementing the identity monad really helped me, because it's just so simple, yet has all the necessary properties of a "real" monad. State was far more difficult, mainly because of the fact that the monad wraps a function, rather than a value (although we know that these are the same thing in Haskell!), but once I cracked this, it really did "all make sense."

The Identity Monad

First, we need a type and type constructor (by Haskell convention I will use the same expression for both, which for this example is "Id").


newtype Id a = Id a deriving Show

Note: the "deriving Show" bit is there just in case I want to print these to the console. It could have just been:


newtype Id a = Id a

Then I'm going to need a function to extract the value from the "Id" thingy:


runId :: Id a -> a
runId (Id x) = x

Simple enough, so far?

Note: this could also have been acheived using Haskell's "record syntax" as follows (saving the trouble of writing the extra function):


newtype Id a = Id { runId :: a } deriving Show

Now, the Monad bit. Don't blink, or you might miss it!


instance Monad Id where
 return a = Id a
 m >>= f = f (runId m)

Identity monad... done!

I'm not going to explain this in any great detail. You've probably read about "return" and "bind (>>=)" elsewhere. In bind: "m" is the monadic value, "f" is a function (a -> mb) and the bit on the right of the equals just says "run f on the value inside of m."

Some functions that operate on Id thingies:


idPlusBar :: Id String -> Id String
idPlusBar x = do
 y <- x
 z <- Id (y ++ "bar")
 return z

This function operates on Id Strings appends "bar" to the value.

Note: the function above could be written more simply as:


idPlusBar x = Id $ (runId x) ++ "bar"

but I included it in long form for illustration purposes.

Here are a couple that operate on Id Ints:


idPlusTwo :: Id Int -> Id Int
idPlusTwo x = Id $ (runId x) + 2

idTimesThree :: Id Int -> Id Int
idTimesThree x = Id $ (runId x) * 3

idComposed :: Id Int -> Id Int
idComposed = idPlusTwo . idTimesThree

Now, here's a function ("main" in this case, as I can run this from the command line), which uses HUnit (you'll need to "import Test.HUnit") to test some assumptions:


main = do
 putStrLn "running tests"
 -- Id String tests
 let x = Id "foo"
 let test1 = TestCase $ assertEqual "String foo" "foo" $ runId x
 let test2 = TestCase $ assertEqual "String foobar" "foobar" $ runId $ idPlusBar x
 -- Id Int tests
 let y = Id 3 :: Id Int
 let test3 = TestCase $ assertEqual "Int 3" 3 $ runId y
 let test4 = TestCase $ assertEqual "Int 5" 5 $ runId $ idPlusTwo y
 let test5 = TestCase $ assertEqual "Int 11" 11 $ runId $ idComposed y
 -- run tests
 let tests = TestList [  TestLabel "foo" test1
       ,TestLabel "foobar" test2
       ,TestLabel "int 3" test3
       ,TestLabel "int 5" test4
       ,TestLabel "int + *" test5
      ]
 runTestTT tests

So far, none of this is too challenging. You may have noticed that there's absolutely no point to any of it, however. That's OK! The idea was just to get used to wrapping stuff up and operating on it. The first version of "idPlusBar" is interesting because it illustrates how things work in "do-notation."

If you've got this monad loaded up into ghci, you can see this, too:


*Main> let x = Id "foo"
*Main> x
Id "foo"
*Main> idPlusBar x
Id "foobar"
*Main> let f x = x >>= \a -> Id (a ++ "bar")
*Main> f x
Id "foobar"

where "f" doesn't use "do", but rather "bind" directly.

Now, if you're still with me, we shall leave the tidy, simple world of identity, and enter the mysterious wonderment that is "state."

The first part of this excercise involved coming up with a super simple case where I needed to have a state parameter of some kind and a bunch of functions that depended on it. I elected to make the State an "Int" and the value the functions operate on a List of Ints. The key is that in order to be sequenced correctly the state would have to be passed from one function to the next. The functions use the state int to transform the lists, and to add complication (and make it easy to prove it's working properly) I will increment the state with each function that gets executed.

As you probably know, you can do it by "threading the state" and not using monads at all. Let's try that first:


addS :: Int -> [Int] -> [Int]
addS s xs = map (+ s) xs

timesSminus1 :: Int -> [Int] -> [Int]
timesSminus1 s xs = map (* (s-1)) xs

addStimes2 :: Int -> [Int] -> [Int]
addStimes2 s xs = map (+ (s*2)) xs

where s::Int is the 'state' and the list is the thing to be transformed

Now we can sequence these functions, and "manually" increment the state:


allS :: Int -> [Int] -> [Int]
allS s xs = let as = addS s xs;
    s' = s+1;
    as' = timesSminus1 s' as;
    s'' = s'+1;
   in addStimes2 s'' as'

It's already easy to see, even with this simple example, that this has the potential to become quite cumbersome. It's also obvious that the incrementing of the state is going to get awfully repetitive and should be abstracted in some way.

Enter the state monad...


type S = Int

newtype St s a = St { runState :: S -> (S, a) }

instance Monad (St s) where
 return value = St $ \state -> (state, value)
 monadSt >>= func = St $ \state->let (state', value) = runState monadSt state
         in runState (func value) (state'+1)

Note: this is not a generic state monad, but one which works exclusively on Ints and increments the state with each function execution (in "bind").

Here are our functions again, using the monad:


addS' :: [Int] -> St Int [Int]
addS' xs = St $ \state -> (state,map (+ state) xs)

timesSminus1' :: [Int] -> St Int [Int]
timesSminus1' xs = St $ \state -> (state,map (* (state-1)) xs)

addStimes2' :: [Int] -> St Int [Int]
addStimes2' xs = St $ \state -> (state,map (+ (state*2)) xs)

allS' :: [Int] -> St Int [Int]
allS' xs = do
    as <- addS' xs
    as' <- timesSminus1' as
    as'' <- addStimes2' as'
    return as''

The last function in particular is quite clean compared to its non-monadic cousin.

Once again, we'll use HUnit to test some assumptions...


main = do
 putStrLn "running tests"
 -- set up some values to use
 let xs = [1,2,3]
 let state1 = 3
 let state2 = 4
 let state3 = 5
 -- threading state tests
 let test1 = TestCase $ assertEqual "increment by s" [4,5,6] $ addS state1 xs
 let test2 = TestCase $ assertEqual "times (s-1)" [3,6,9] $ timesSminus1 state2 xs
 let test3 = TestCase $ assertEqual "increment by s*2" [11,12,13] $ addStimes2 state3 xs
 -- put it together in allS
 let test4 = TestCase $ assertEqual "the lot" [22,25,28] $ allS state1 xs
 -- state monad tests
 let test1' = TestCase $ assertEqual "m increment by s" [4,5,6] $ snd $ (runState $ addS' xs) state1
 let test2' = TestCase $ assertEqual "m times (s-1)" [3,6,9] $ snd $ (runState $ timesSminus1' xs) state2
 let test3' = TestCase $ assertEqual "m increment by s*2" [11,12,13] $ snd $ (runState $ addStimes2' xs) state3
 -- monadic state passing in allS'
 let test4' = TestCase $ assertEqual "m the lot" [22,25,28] $ snd $ (runState $ allS' xs) state1
 -- run tests
 let tests = TestList [  TestLabel "test1" test1
       ,TestLabel "test2" test2
       ,TestLabel "test3" test3
       ,TestLabel "test4" test4
       ,TestLabel "m test1'" test1'
       ,TestLabel "m test2'" test2'
       ,TestLabel "m test3'" test3'
       ,TestLabel "m test4'" test4'
       ]
 runTestTT tests

The real eye-opening thing here is that allS' doesn't take a state parameter at all. That's because allS' is returning a state monad (a function from state to state,value). So, we pass the state to the function (or value) which is returned by this function. This is the trick... and it is a little bit like magic!

So,

allS' takes the list ("xs") and returns the monad. runState extracts the function from the monad (\s -> (s,a)), and we pass the initial state to that. That in turn returns the state,value pair, so we take the "snd" value of the tuple.

This is it again:


snd $ (runState $ allS' xs) state1

Nice, huh?

So, our function doesn't have to mess around with state at all. It's all handled in the monad's bind function, including the incrementing of the state's value with each function call.

Once again, I would just add that the most striking thing in all this is its simplicity (a trait I've come to expect from Haskell). It seems that monads weren't that complicated after all.

Now, what are these Monad Transformer things?

Automated Unit Testing with ScalaCheck

2010-09-09T21:48:00.002+01:00

It just sounds like a great idea, doesn't it? Automated testing? Fantastic! I can see the tagline now: Say Goodbye to Technical Debt Forever!

Before I begin, let's establish what we're talking about: What do we really mean when we say automated testing? In this case, what I do not mean is "test automation". I'm not talking about running tests automatically, by, say, using Continuous Integration. I'm talking about test generation. Test generation means, if you've got a function that operates on a particular input, you specify a "property" (or several properties) about that function, and then "generators" will give you variations on the input data to test all the edge cases, etc., and tell you whether the property holds. You can use default generators for known datatypes, or define your own.

I've been interested in the idea of automated testing for some time, having come across QuickCheck (a Haskell library). Please see this quite interesting interview with John Hughes (one of QuickCheck's creators) on Functional Programming.

I don't know Haskell very well, and have done more with Scala to date, so I thought I'd have a look at ScalaCheck to satisfy my curiosity. The following represents a quick walk through of the basics of ScalaCheck.

Please note: I'm using Scala 2.7.7 and ScalaCheck 1.6, and I'll do all this via the interpreter.

So, on the command line...

scala -cp scalacheck_2.7.7-1.6.jar
scala> import org.scalacheck.Prop

scala> Prop.forAll( (a:Int, b:Int) => a+b == b+a ).check         
+ OK, passed 100 tests.

Nice. I just generated a hundred tests. This TDD business is going to be a breeze! But what was really going on here?

"forAll" is a method of the Prop ("property") object that returns a Prop instance. The parameter in this case is a function which is implicitly converted into a Prop. That Prop's "check" method is called and the result of the property check is displayed.

What was the property under test? For any given two Ints, if they are added together, the result of that addition will be equal regardless of the order in which they are added.

Admittedly, not very interesting, but nonetheless, this example illustrates a number of things, not least of which is that ScalaCheck generated, by way of its default Int generator (or Gen), 100 tests which all passed.

Here's a longer way to write the same thing which may make some of this clear:

scala> val f = (a:Int, b:Int) => a+b == b+a
f: (Int, Int) => Boolean = 

scala> val p = Prop.forAll(f)
p: org.scalacheck.Prop = Prop

scala> p.check
+ OK, passed 100 tests.

Great.

And you can easily combine multiple properties. Let's say you have 3 Props (p1,p2,p3) the first two of which are correct and the last is not:

scala> (p1 && p2).check
+ OK, passed 100 tests.

scala> (p1 && p3).check
! Falsified after 1 passed tests.
> ARG_0: T(8,true)

scala> (p1 || p3).check
+ OK, passed 100 tests.

Now, if you're like me, and you really want to know what's going on, note that it is possible to "collect" the data that is generated for the tests.

For example...

scala> val p = Prop.forAll((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a })
p: org.scalacheck.Prop = Prop

This is essentially the same thing, but notice the insertion of the call to "Prop.collect".

Now when we check it, we get this...

scala> p.check
+ OK, passed 100 tests.
> Collected test data:
4% (0,0)
1% (-22,38)
1% (4,13)
1% (50,30)
1% (-1,44)
1% (4,28)
1% (12,-16)
1% (-1,-16)
1% (-11,2147483647)
1% (72,-17)
1% (8,-1)
1% (-3,-77)
1% (-46,-19)
1% (-23,-1)
1% (-26,-22)
1% (-21,46)
1% (-48,48)
1% (1,-13)
1% (18,1)
1% (19,-1)
1% (34,-3)
1% (-1,-29)
1% (-58,-63)
1% (1,15)
1% (4,11)
1% (17,-22)
1% (-8,-8)
1% (3,-15)
1% (-9,-36)
1% (-92,78)
1% (28,-42)
1% (-18,0)
1% (3,1)
1% (4,2147483647)
1% (-2147483648,10)
1% (37,-2147483648)
1% (8,-2147483648)
1% (21,0)
1% (1,39)
1% (17,88)
1% (0,-48)
1% (38,-44)
1% (0,6)
1% (2147483647,29)
1% (-1,-24)
1% (-28,11)
1% (-4,46)
1% (28,-63)
1% (35,8)
1% (9,12)
1% (18,29)
1% (8,-18)
1% (-3,-1)
1% (0,14)
1% (47,1)
1% (4,3)
1% (-84,-7)
1% (13,31)
1% (-60,29)
1% (-41,-43)
1% (-75,67)
1% (-3,2)
1% (-73,-57)
1% (-10,-13)
1% (15,-3)
1% (2147483647,-25)
1% (8,0)
1% (-79,76)
1% (48,3)
1% (-5,9)
1% (4,0)
1% (-2147483648,8)
1% (43,-25)
1% (-46,54)
1% (-9,-2147483648)
1% (52,33)
1% (-10,4)
1% (36,-20)
1% (1,0)
1% (30,1)
1% (-2147483648,22)
1% (-27,-12)
1% (0,1)
1% (-58,2147483647)
1% (38,-27)
1% (-38,19)
1% (23,38)
1% (11,-15)
1% (-2147483648,17)
1% (-17,51)
1% (-36,-59)
1% (1,14)
1% (-3,-11)
1% (-31,29)
1% (-60,-44)
1% (2147483647,1)
1% (4,16)

So, we can see what's at work behind ScalaCheck's default generator (Gen) for "Int". A lot of random numbers there. Edge cases, positive and negative, zero, etc. Thanks, ScalaCheck! You just saved me a lot of test writing!

Let's look at a couple of things you can do with "Gen".

First, there's a "choose" method to choose amongst options. Say, we only wanted to check our property with Ints between 1 and 100.

scala> Prop.forAll(Gen.choose(1,100),Gen.choose(1,100))((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a }).check
+ OK, passed 100 tests.
> Collected test data:
1% (32,48)
1% (80,30)
1% (26,31)
1% (15,91)
1% (39,3)
1% (12,68)
1% (93,71)
1% (8,22)
1% (44,58)
1% (4,73)
1% (76,96)
1% (30,98)
1% (55,100)
1% (76,28)
1% (93,13)
1% (46,47)
1% (78,56)
1% (77,30)
1% (69,17)
1% (29,81)
1% (11,39)
1% (40,3)
1% (17,56)
1% (2,81)
1% (31,2)
1% (42,7)
1% (47,22)
1% (48,17)
1% (79,33)
1% (48,40)
1% (52,83)
1% (95,50)
1% (63,30)
1% (21,96)
1% (29,30)
1% (72,33)
1% (70,56)
1% (24,79)
1% (68,16)
1% (89,7)
1% (82,21)
1% (13,35)
1% (73,88)
1% (99,25)
1% (79,8)
1% (52,84)
1% (92,32)
1% (90,77)
1% (88,60)
1% (51,4)
1% (32,93)
1% (62,90)
1% (9,14)
1% (40,36)
1% (78,80)
1% (84,5)
1% (31,63)
1% (7,14)
1% (48,88)
1% (92,21)
1% (31,52)
1% (58,2)
1% (69,82)
1% (84,7)
1% (19,69)
1% (48,94)
1% (50,16)
1% (59,52)
1% (63,63)
1% (55,40)
1% (50,39)
1% (30,26)
1% (15,61)
1% (85,30)
1% (93,90)
1% (19,49)
1% (5,61)
1% (28,56)
1% (41,60)
1% (88,82)
1% (40,62)
1% (96,48)
1% (85,96)
1% (88,7)
1% (1,91)
1% (14,82)
1% (1,56)
1% (24,98)
1% (86,41)
1% (77,65)
1% (4,38)
1% (78,77)
1% (4,62)
1% (84,36)
1% (90,33)
1% (12,67)
1% (45,45)
1% (54,75)
1% (38,94)
1% (14,91)

Or, we can see that the range is inclusive.

Or, we can define a Gen which is based on conditional statements using "suchThat".

val smallOdds = Gen.choose(1,100) suchThat (_ % 2 == 1)
smallOdds: org.scalacheck.Gen[Int] = Gen()

And use that for both our Ints:

scala> Prop.forAll(smallOdds,smallOdds)((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a }).check
+ OK, passed 100 tests.
> Collected test data:
2% (19,75)
2% (53,55)
1% (35,47)
1% (15,75)
1% (13,41)
1% (51,99)
1% (39,23)
1% (97,35)
1% (47,99)
1% (79,87)
1% (95,55)
1% (45,49)
1% (3,61)
1% (73,93)
1% (13,5)
1% (45,45)
1% (31,51)
1% (61,9)
1% (55,55)
1% (25,47)
1% (99,21)
1% (65,29)
1% (47,49)
1% (47,89)
1% (95,39)
1% (43,73)
1% (39,49)
1% (41,17)
1% (89,25)
1% (25,39)
1% (37,27)
1% (17,49)
1% (77,37)
1% (11,9)
1% (15,13)
1% (37,47)
1% (93,77)
1% (5,75)
1% (19,87)
1% (35,39)
1% (3,21)
1% (65,53)
1% (73,83)
1% (39,63)
1% (31,53)
1% (69,31)
1% (99,65)
1% (9,97)
1% (55,57)
1% (83,71)
1% (41,35)
1% (59,69)
1% (3,55)
1% (85,9)
1% (87,71)
1% (33,33)
1% (9,83)
1% (59,55)
1% (19,51)
1% (49,45)
1% (95,61)
1% (45,21)
1% (57,61)
1% (87,89)
1% (35,71)
1% (85,89)
1% (61,91)
1% (47,5)
1% (3,69)
1% (3,17)
1% (39,85)
1% (95,89)
1% (87,11)
1% (85,49)
1% (55,79)
1% (7,97)
1% (93,29)
1% (37,61)
1% (51,19)
1% (67,53)
1% (71,9)
1% (57,11)
1% (41,85)
1% (31,27)
1% (31,57)
1% (87,49)
1% (69,53)
1% (59,77)
1% (23,87)
1% (29,21)
1% (11,59)
1% (69,69)
1% (77,79)
1% (85,65)
1% (31,31)
1% (97,15)
1% (11,15)
1% (75,85)

Let's say you wanted to generate data for an arbitrary case class. ScalaCheck comes with an "Arbitrary" class to do just that:

scala> case class T(a:Int, b:Boolean)
defined class T

scala>import org.scalacheck.Arbitrary
import org.scalacheck.Arbitrary

scala> val genT = for { a <- Gen.choose(1,10); b <- Arbitrary.arbitrary[Boolean] } yield T(a,b)
genT: org.scalacheck.Gen[T] = Gen()

So, our class should be comprised of an Int value 1 to 10 (inclusive) and a boolean.

scala> Prop.forAll(genT)((t:T) => Prop.collect(t) { t.a >= 1 && t.a <= 10 && (t.b == true || t.b == false) }).check
+ OK, passed 100 tests.
> Collected test data:
9% T(3,true)
8% T(10,true)
7% T(6,false)
7% T(2,true)
6% T(4,false)
6% T(7,false)
6% T(2,false)
6% T(1,true)
6% T(4,true)
5% T(5,false)
5% T(8,true)
5% T(8,false)
5% T(3,false)
4% T(5,true)
4% T(7,true)
3% T(10,false)
3% T(1,false)
3% T(9,false)
1% T(9,true)
1% T(6,true)

Lovely.

Another interesting feature is the ability to classify values that are chosen by the generator. For example...

scala> Prop.forAll(genT)(t => Prop.classify({ t.b==true }, "b is true")(p)).check
+ OK, passed 100 tests.
> Collected test data:
60% b is true

Note: "p" is a property which holds in this case.

Labels:

Let's say you have three properties you are testing, and two of the three are correct...

scala> val p1 = Prop.forAll(genT)((t:T) => t.a >= 1)
p1: org.scalacheck.Prop = Prop

scala> p1.check
+ OK, passed 100 tests.

scala> val p2 = Prop.forAll(genT)((t:T) => t.a <= 10)
p2: org.scalacheck.Prop = Prop

scala> p2.check
+ OK, passed 100 tests.

scala> val p3 = Prop.forAll(genT)((t:T) => t.b == false)
p3: org.scalacheck.Prop = Prop

scala> p3.check
! Falsified after 2 passed tests.
> ARG_0: T(1,true)

Run in combination, it's impossible to know which property failed:

scala> (p1 && p2 && p3).check
! Falsified after 0 passed tests.
> ARG_0: T(2,true)

So, you can label the individual properties using a special ":|" operator:

scala> (p1 :| "a greater than or equal to 1" && p2 :| "a less than or equal to 10" && p3 :| "b always false").check
! Falsified after 0 passed tests.
> Labels of failing property:
b always false
> ARG_0: T(8,true)

Lastly, another cool feature which is very "scala-ish": implicit generator definitions. Basically, ScalaCheck uses implicit definitions to decide which generator to use. By default, there are implicit generators for basic types (Int, String, etc., as seen above where no generator definition was necessary for Ints). However, here's how implicit def works for our case class (T):

scala> implicit def arbT: Arbitrary[T] = Arbitrary(genT)
arbT: org.scalacheck.Arbitrary[T]

scala> Prop.forAll((t:T) => t.a >= 1).check       
+ OK, passed 100 tests.

Don't blink or you'll miss the magic!

This is all gone into in much more detail in the ScalaCheck User Guide, but I hope this overview will prove helpful to some.

Developing a Project Projection Chart in Excel When the Backlog Contains Sub-Projects (Agile/Scrum)

2010-08-21T14:45:00.004+01:00

With the keyword being "Excel". (Oh, it's a particular beast, alright.)

Lest I be burned at the stake by a ferocious pack of Excelites, I begin with the following admission: I am a newbie when it comes to Excel. I write this blog partly to get constructive feedback (should anyone have any), and partly to help anyone who might struggle with a similar problem and who might receive some modicum of joy from this method.

The Probelm

First, let me define a problem: You have a backlog that contains work which is categorized in some way. The categories might be "sub-projects", "feature sets", or similar. You want to get a picture of when a particuar category is going to be worked on, or when work overall on a particuar category is going to be done. Furthermore, you don't just want to know, you want to be able to make a big, visible chart out of the data.

This excercise was in part inspired by this blog post: Is the Gantt Chart Useless in Agile Projects? written by Michael Cardus on Jurgen Appelo's Management 3.0 site. That post deals specifically with Gantt charts in Agile, and a Gantt chart wasn't exactly what we needed, but is quite similar. So the "literature", so to speak, on doing Gantt charts in Excel should prove useful.

Let's say your backlog looks like this:

So, we have four categories with various stories which have been put in a completely random (uh, I mean very sensible and thought-out) order by the Product Owner. Each story has a size. We also (and maybe not everyone does this) keep old backlog items around and mark as "done", so we'll need to take that into consideration.

Let's also assume that using Excel is a requirement. Please don't question this requirement; I did, and my arse still hurts from the spanking.

The first thing I did when trying to "solve" this problem was to try a load of various ways of summarizing the data and visualizing it with built in Excel chart tools. Once I managed to reinsert all the hair that I had pulled out of my head (actually, there's still a spot on top which isn't covered and I'm pretty sure that wasn't there before ;-), I eventually settled on this method: Convert the story points to days based on velocity, calculate the work activity on a per day basis, then use a stacked bar chart to visualize.

Let's look at those steps individually:

Step 1: Points to Days

Now, I know what you're thinking: story points are not a measure of time! Yes, but that doesn't mean to say that we're not allowed to use our average (and worst case) velocity to make loose predictions for release planning purposes. Work with me here, I'm pretty sure I'm not violating some law of Scrum physics that's going to open up a hole in the Points/Time Continuum.

The way I did this conversion was to use a few hidden columns in the backlog to show "preceding points" for each story (i.e. story points up to this story in the backlog which were not yet complete), as well as "days before" and "days at completion" both based on velocity.

The formula for getting the preceding points was a bit tricky. To sum the "undone" work in the backlog that precedes this story:

{=SUM(IF(($E4<>"y")*($D4>0)*($E$4:$E$14<>"y")*(ROW($D$4:$D$14)<ROW()),$D$4:$D$14,0))}

Where D is the Points column, and E is the Done? column.

It's an array formula (a key fact: look up array formulas if you don't know about them, they're quite cool) that basically looks at the whole table to get points for stories where "Done?" is not "y" and the row number is less than the current row number. Looks easy in hindsight, doesn't it!

Note that the "*" is how we "AND" conditions in an array formula. Excel's "AND" function won't work here.

The next two calculations really are easy.

Calculate the number of days until this story will start (based on current velocity):

=($F4/($C$23/7))

Calculate the number of days until this work will be complete (based on current velocity and story size):

=IF($E4<>"y",($F4+$D4)/($C$23/7),0)

Where F is the "preceding points" column we calculted above, C23 is the average sprint velocity and 7 is the number of days per sprint.

Now, with those columns unhidden, the backlog looks like this:

So, with that chore out of the way, let's talk about what we want this graph to look like. I always find with Excel that it's easier to think about what data you need to create a workable chart first, and then format the data accordingly. It's too easy to disappear down a rabbit hole if you take what seems to be sensible data and try to make the chart fit *it*.

Ultimately, this is what we end up with (what we are going for):

The Gantt chart literature made it quite clear that a stacked bar chart would be our friend. I played around a bit with line charts and scatter plots, but nothing worked quite right. One thing that slightly bugs me about Excel is not being able to do a chart which is categorical on both axes. If that were possible then a mapping of our categories to a Sprint date (or Sprint number) would be relatively easy. I'm sure there's a mathematical reason for it (surely there's a reason for everything Microsoft does?), but Excel only supports value/value (scatter plot) and category/value (everything else).

Step 2: The Chart Data

Now this is where it gets ugly. How the hell are we going to get a chart that looks like that from our data? Turns out, it's pretty easy if we just stop thinking about it and lay the data out in the dumbest way we can think of! (I've always had this theory that computers are really, really dumb, and programmers are just people who are good at breaking stuff down into simple instructions that make sense to a dummy!)

Take a look at the picture below to see how this was done. Better yet, download the spreadsheet here.

Note: the pic doesn't show the whole thing. There is data for every day that we want to show (42 days, or 6 sprints of 7 days, in this case).

So, literally, there's a column for every day, and there's a cell for every category/day that has a 1 or 0 depending on whether there is work in that category for that day. The formula looks like this:

Determine if this Category on this Day contains work (show 1 or 0):

{=SUM(IF((Table1[category]=$U5)*(Table1[done?]<>"y")*(V$3>=Table1[days to start])*(V$3<Table1[days to finish]),1,0))}

Where U is the Category column, and V3 points to the day number of the current column.

So, basically, for all items in this category: is this day in between the start and finish day of these pieces of work?

The "1" actually creates the "height" for our bar in the graph. The trick here is the use of the inverse items (blanks) to fill in the vertical space below other categories. The "blanks" are just set to "No Fill" in the chart, so they are literally blanks.Figuring that one out was my proudest moment using Excel to date (OK, you caught me, I'm easily impressed).

A few other bits of trickery: eliminate gaps in bars so they look continuous; set horizontal axis major unit to 7 for sprint dates; delete left axis; delete "blanks" from legend; and we're done!

So, please download that spreadsheet and use this if you like it. Or, if you have a better idea, I'd love to hear it!

'Bye for now...

What's So Funny About ATDD, BDD, and Dependency Injection?

2010-06-08T22:52:00.014+01:00

With apologies to Elvis Costello...

So, I've been thinking about a way to introduce the concepts of Acceptance Test Driven Development (ATDD) and Behavior Driven Development (BDD) to our developers. I'd also like to cover Dependency Injection as this seems to be something that comes up a lot when we discuss TDD in our little neck of the woods.

As an example, I've used an abstraction from a large project that we've been working on. The project is a data conversion project. I won't go into details, but the gist is this: take some data (probably from a database), perform some validation checks on it, filter stuff that we don't want, convert some stuff that we do want, and send it to some kind of output stream.

Our project is Java-based, but I've chosen to use Scala for this example for a number of reasons:
1. Scala seemed interesting, and I wanted to learn more.
2. Scala has parser combinators which is perfect to test my idea of using a DSL to define our conversion rules.
3. ScalaTest supports ATDD and BDD beautifully via its FeatureSpec and FlatSpec traits, respectively.
4. Scala has some neat ways of dealing with dependency injection (see Chapter 27 of the Odersky book).
5. Scala is fully interoperable with Java, so we can reuse any libraries or code from the existing project.

See Scala and ScalaTest here and here, respectively.

So, let's get started... (please bear in mind that I am well aware that my Scala code, being a relative newcomer to it, is likely to be a bit rough... I welcome suggestions!)

Firstly, I set up an ant build.xml that would compile my Scala code and run my ScalaTest tests. There are basically 3 files (not counting the build.xml): 1) test/UnitTests.scala, 2) test/AcceptanceTests.scala, and 3) src/Conversion.scala. It's a pretty small project for purposes of this exercise, so there's no need to get crazy here, plus it's nice to be able to easily ":load" files in the Scala interpreter (note: by the end it was just approaching large enough to warrant breaking some stuff out into more files, but I chose not to go there).

Note: the code for this project can be downloaded here. Please take a look. I am just going to focus this blog on a few of the more interesting points.

The application design was set up with the idea of ATDD in mind. There would be a Conversion class that would take all the necessary bits as parameters: those bits being the datasource, the rules for the conversions, and some kind of output stream device. So, let's look at some of the acceptance tests first.

The first thing to bear in mind was that I was going to need some kind of Data type for retrieving and carrying the input data. I knew that this might come from a database, but I wanted to be able to "inject" it for purposes of these tests. I decided to go with a simple Map[String,String] to capture field names and values. There would be an abstract class called "Data" and one called "DataSet" that I could implement to get my tests set up. The DataSet class would have a foreach method, so that rows could be read from a database one at a time for processing (rather than pass a whole set of data at once, as our datasets are quite large in reality).

Additionally, I would need "Rules". I figured that these would be read from a file, and would comprise our "DSL" bit, so I opted to have these be simple Strings. Again, an abstract class called RuleSet (List[String] for testing) would also have the potential for some other functionality, such as reading the Rules from a file.

I also used a static "Log" object to capture system output for monitoring. The default Log would just write to STDOUT, but I also implemented a version that would write to a list buffer, so that I could capture that stuff for testing, too.

Also, an Output class which I implemented as a ListBuffer as well for testing. In reality, this Output class could write to a database, or send data across the network, etc.

Lastly, a Conversion object which takes each of our little dependencies via its apply method, and does it's magic (basically apply all Rules to each Data object in the DataSet and capturing the output via our Output object).

Here's a snapshot of the acceptance tests...

class RuleApiFeatureSpec extends FeatureSpec with GivenWhenThen with MustMatchers {

 val d1 = MapData(Map(("id","1"),("a","5foo"),("b","1234bar234")))
 val d2 = MapData(Map(("id","2"),("a","foo"),("b","bar45")))
 val d3 = MapData(Map(("id","3"),("a","454foo45"),("b","445bar")))
 val d4 = MapData(Map(("id","4"),("a","43foo45"),("b","bar87")))
 val d5 = MapData(Map(("id","5"),("a","foo45"),("b","3bar4546")))
 val d6 = MapData(Map(("id","6"),("a","cfoo"),("b","bar")))

 object MapDataSet extends DataSet {
  // in production, DataSet may come from a database (can be read one record at a time in foreach)
  val data = List(d1,d2,d3,d4,d5,d6)
     def foreach(f:(Data)=>Unit):Unit = { data.foreach(f) }
 }

 val r0 = "f:b /^[0-9][0-9]+.*/ Convert x:transformAandB" // convert a and b if b starts with two numbers
 val r1 = "f:b /^[0-9]+.*/ Convert x:transformA" // convert a if b starts with a number
 val r2 = "f:a /^[0-9]+.*/ Filter" // filter if a starts with a number
 val r3 = "f:b /^b.*/ Convert x:transformB" // convert b if b starts with "b"
 val r4 = "f:a /^c.*/ NoMatch" // bad rule
 
 val rules = RuleList(List(r0,r1,r2,r3,r4),MyRuleParser)

 class ListOut extends Output {
     var container = new ListBuffer[Data]()
     def apply(dataIn:Data) = {
        container+dataIn
     }
  def getById(n:String):Data = { container.filter( (d) => d("id") == n )(0) } // assuming one match here
     override def toString = "***OUTPUT***\n"+container.mkString("\n")
 }

 Log.setType("list")
 
 feature("Conversion API"){
  scenario("Filtered Output"){
   given("The default testing scenario set up above")
   when("I run the conversion")
   val out = new ListOut
   Conversion(MapDataSet,rules,out)
   then("I expect to see five records in the output")
   out.container must have length 5
   then("The filtered record should be id 4")
   out.container.map( d => d("id") ).contains("4") must be === false
   then("The records not filtered would be 1,2,3,5,6")
   out.container.map( d => d("id") ).sameElements(List("1","2","3","5","6")) must be === true
  }
 }
}

So, this system is geared toward Acceptance Testing, via dependency injection. And ScalaTest's FeatureSpec is great for this. It even gives you a nice "Given When Then" syntax, so that your real world acceptance criteria can easily be translated into automated acceptance tests. You are writing acceptance criteria, aren't you!?!? ;-)

Other acceptance tests follow more or less the same pattern.

BTW, Here's the code for the Conversion object:

object Conversion extends Logging {
    def apply(data:DataSet, rules:RuleSet, output:Output) = {
        data.foreach(
            d => {
                log("PRE PROCESSING: "+d)
                val o = rules(d)
                log("POST PROCESSING: "+o)
                if(!o.isFiltered()) { log("ADDING TO OUTPUT: "+o); output(o); }
            }
        )
    }
}

The foreach method on the DataSet is called and the rules applied to each resulting object. So, by implementing a "DbDataSet", for example, these records could be pulled from a database.

I'll just show the RuleSet code, because I love this little Scala trick:

case class RuleList(rs:List[String],rp:RuleParser) extends RuleSet with Logging {
    val rules = rs.map { r => rp.parse(r) }.filter{ r => r != None }.map{ r => r.get }.toSeq
    def apply(dataIn:Data):Data = {
        log("APPLYING RULES")
        Function.chain(rules)(dataIn)
    }
}

I suspect there might be a cleaner way to do that map/filter/map thing, but the cool thing is the line: Function.chain(rules)(dataIn). Basically, with the "rules" being implementations of Function[Data,Data], it's basically saying "apply this data object to this chain of Rule functions". Nice.

This line sets up the rules, and references the RuleParser:

val rules = RuleList(List(r0,r1,r2,r3,r4),MyRuleParser)

Here's the code for the RuleParser:

object MyRuleParser extends RegexParsers with RuleParser {
    def fieldStart:Parser[String] = "f:"
    def field:Parser[String] = "[A-Za-z]+".r ^^ { _.toString }
    def functionStart:Parser[String] = "x:"
    def function:Parser[String] = "[A-Za-z]+".r ^^ { _.toString }
    def pattern:Parser[Regex] = "/.*?/".r ^^ { case p => new Regex("/".r.replaceAllIn(p,"")) }
    def action:Parser[String] = "Filter|Convert".r ^^ { _.toString }
    def rule1:Parser[Rule] = fieldStart~field~pattern~action~functionStart~function ^^ { case fs~f~p~a~xs~x => Rule(new PatternMatcher(f,p),a,ConvertFuns.functions(x)) }
    def rule2:Parser[Rule] = fieldStart~field~pattern~action ^^ { case fs~f~p~a => Rule(new PatternMatcher(f,p),a,NoConvert) }
    def rule:Parser[Rule] = rule1 | rule2
    def parse(input:String):Option[Rule] = parseAll(rule,input) match {
        case Success(e,_) => Some(e)
        case f: NoSuccess => None
    }
}

This uses parser combinators to turn our "Rule" (strings) into "Rule" objects. Rule itself looks like this:

case class Rule(matcher:MatchFun, action:String, conversion:ConvertFun) extends Function[Data,Data] with Logging {
    def apply(dataIn:Data):Data = {
        log("RULE: "+this.action+" on data "+dataIn+":")
  if(dataIn.isFiltered){ log("FILTERED"); dataIn }
  else if(matcher(dataIn)){
   action match {
    case "Filter" => { log("FILTERING"); dataIn.setFiltered() }
    case "Convert" => { log("CONVERTING"); conversion(dataIn) }
    case _ => { log("BAD ACTION"); dataIn }
   }
  }
  else{ log("NO MATCH"); dataIn }
    }
}

Now, I'd like to talk about unit tests.

For unit tests, I chose to use the FlatSpec trait in ScalaTest. The reason is because I wanted to do BDD, with the basic "Object when something should something in...{ code }" type syntax, but I really didn't like all the nesting that goes on in most libraries. Bill Venners has solved that problem for us with FlatSpec. The name is as it implies. Here are some examples...

class MapDataSpec extends FlatSpec {
 val m = MapData(Map(("id","1")))
 "A MapData object with an 'id' field" should "have an 'id' key" in {
  expect(true){ m.has("id") }
 }
 "A MapData object with an 'id' field with value '1'" should "return that value for that key" in {
  expect("1"){ m("id") }
 }
 "A MapData object which has no 'a' field set" should "throw error when accessed" in {
  expect(false){ m.has("a") }
  intercept[NoSuchElementException]{ m("a") }
 }
 "A MapData object which has a field set" should "recognize that key and return that value for that key" in {
  val m2 = m.set("a", "foo")
  expect(true){ m2.has("a") }
  expect("foo"){ m2("a") }
 }
 "A MapData object which has not had filtered set" should "return false for isFiltered" in {
  expect(false){ m.isFiltered }
 }
 "A MapData object which has filtered set" should "return true for isFiltered" in {
  val m2 = m.setFiltered
  expect(true){ m2.isFiltered }
 }
}

And the unit tests continue on like that (please see linked code for details). Nice, simple syntax. Very intuitive.

I hope this makes some sense, and proves helpful to some.

Edit: BTW, I have found a slightly more elegant way of doing this:

val rules = rs.map { r => rp.parse(r) }.filter{ r => r != None }.map{ r => r.get }.toSeq

which is this:

val rules = rs.map { r => rp.parse(r) }.flatMap{ x=>x }.toSeq

On the Fence: API vs DSL for programming interactive music in the DOME

2010-03-09T02:03:00.006Z

I've been working on an "online" version of the DOME, which is an interactive music engine for video games that I've been developing on and off for a number of years (for more info see Dometechnics). By "online" I mean a version of the engine that could be embedded in a web page, so that it could be made available for developers of Flash and Javascript games.

The engine itself is a Java applet, which is scriptable via Javascript. Java, for all its faults, is still the best platform for producing MIDI-based synthesized music in a web page. Thanks for nothin', Adobe. Keeping an eye on Google's NativeClient, though.

This blog post will take a look at the challenges of getting Javascript and Java to talk to one another in a web page, but mostly, I want to look at the design decisions that I've been making about the API for doing music programming and whether or not it would be more appropriate to call it a "DSL" (although the distinction might just be syntactic sugar).

Lastly, I've embedded a screencast (see above) to demo the concepts discussed here.

In order to make sense of this post, I should explain a little bit about how the DOME works. It's based on the idea of creating a "score" in the sense of a theme and variations on that theme, followed perhaps by another theme. Loosely, it's designed to enable "cinema-like" scores in interactive contexts: "Films often have different themes for important characters, events, ideas or objects, taking the idea from Wagner's use of leitmotif. These may be played in different variations depending on the situation they represent, scattered amongst incidental music." (source Wikipedia).

It sort of looks like this (from Javascript):

dome.setTheme( _JSON-formatted-theme-data_ );
dome.play();
var minor_variation = 'E->Eb,B->Bb|!bar:4'; // C major to C minor, but not in bar 4
dome.bind( _some-dom-element_, 'onclick', function(){ dome.vary(minor_variation) } );

See how easy!?!

The interesting part in terms of this API vs DSL discussion is the line where the variation is defined. Because of limitations in passing anything other than strings or numbers from Javascript to Java, we might have ended up with something like this:

dome.vary('pitch', '+', 5); // not a very attractive looking command to use (and operators as strings? ...bleh)

Instead, we can do this:

dome.vary('pitch+5'); // better. now it's like a mini-script (will require some parsing magic, though)

...Great. (gratuitous Fast Show reference)

That's all well and good, but in fact, the API approach works better in certain situations. Namely when there's a lot of data to pass (for instance, when the initial theme is established), because an API call can just take a JSON-ified Javascript object as a string. That's because we can take advantage of existing libraries for parsing JSON. But having a DSL for creating variations, in this case, allows for the music programming to occur in a more natural-language type of way.

"The key point is that at each layer the API/DSL should allow the user ... to express the intent of what they want to do as easily as possible." (http://www.testdrivensoftware.com/?p=85)

As a side note, if you're the type of person who gets a kick out of making fun of DSL fan boys, I would direct you to chromatic's amusing discourse on the topic.

So, I've got an API to dump a Javascript object on Java and a DSL for scripting that object, once it's deserialized and appropriately instantiated in Java. In a sense this is still API, because it's essentially a function call on an object, passing in a string. There is no interpreter in the traditional sense of a scripting language. But because the string format for defining variations is a mini-language, it kind of becomes a DSL.

...Great.

Another option might have been to use Java 1.6's scripting API (javax.script), but I had made the decision early on that I wanted this system to be compatible with older versions of Java. I believe the difference is that with Java 1.6, one could theoretically write Javascript code and have that evaluated in the Java application (thereby allowing Javascript functions to manipulate the Java objects). It's a nice idea, but, not critical, and Java 1.5 (and previous) compatibility seemed more important. Why? Because at the time of this writing, the MIDI functionality in Java 1.6 on Mac OS X is broken... that's good enough for me!

Another fun point I'd like to make: At one point during the development of this API/DSL, I fell afoul of a little difference between scripting an applet via the Rhino shell and doing it in a web page. In Rhino, you can return a Java object to Javascript from a method in the applet, and pass that object back to the applet in another function call. Not so via LiveConnect!

For example:

var applet = document.MyApplet;
var obj = applet.getSomeObject(); // return a Java object
applet.doSomethingWithObject(obj); // OK in Rhino! Will cause 40 days and nights of floods if you try this in a browser!

So, one kind of work around, is to have Java use a method on the object to execute the relevant code. This could involve having to share state between the applet and the object in question.

var applet = document.MyApplet;
var obj = applet.getSomeObject(); // return a Java object
obj.doSomethingWithMe(); // works OK in a browser

...Great.

So, let's summarize: Embed the DOME in a web page as an applet and make it "scriptable" via Javascript, and support Java 1.5 for unbroken MIDI. With the goal being the ability to create a theme and define variations on that theme, and invoke those variations by binding them to Javascript events (hence making interactive music possible in Flash and Javascript based games). Done! Alright, not so much done, but it is a working prototype.

Because this is not really ready for prime time, I have produced the following screencast to demonstrate the ideas discussed in this blog. I am actively looking for development resource and industry contacts to bring this technology to fruition. Please come back to follow further progress.