Before I begin, let's establish what we're talking about: What do we really mean when we say automated testing? In this case, what I do not mean is "test automation". I'm not talking about running tests automatically, by, say, using Continuous Integration. I'm talking about test generation. Test generation means, if you've got a function that operates on a particular input, you specify a "property" (or several properties) about that function, and then "generators" will give you variations on the input data to test all the edge cases, etc., and tell you whether the property holds. You can use default generators for known datatypes, or define your own.
I've been interested in the idea of automated testing for some time, having come across QuickCheck (a Haskell library). Please see this quite interesting interview with John Hughes (one of QuickCheck's creators) on Functional Programming.
I don't know Haskell very well, and have done more with Scala to date, so I thought I'd have a look at ScalaCheck to satisfy my curiosity. The following represents a quick walk through of the basics of ScalaCheck.
Please note: I'm using Scala 2.7.7 and ScalaCheck 1.6, and I'll do all this via the interpreter.
So, on the command line...
scala -cp scalacheck_2.7.7-1.6.jar
scala> import org.scalacheck.Prop
Nice. I just generated a hundred tests. This TDD business is going to be a breeze! But what was really going on here?
"forAll" is a method of the Prop ("property") object that returns a Prop instance. The parameter in this case is a function which is implicitly converted into a Prop. That Prop's "check" method is called and the result of the property check is displayed.
What was the property under test? For any given two Ints, if they are added together, the result of that addition will be equal regardless of the order in which they are added.
Admittedly, not very interesting, but nonetheless, this example illustrates a number of things, not least of which is that ScalaCheck generated, by way of its default Int generator (or Gen), 100 tests which all passed.
Here's a longer way to write the same thing which may make some of this clear:
scala> val f = (a:Int, b:Int) => a+b == b+a
f: (Int, Int) => Boolean =
scala> val p = Prop.forAll(f)
p: org.scalacheck.Prop = Prop
scala> p.check
+ OK, passed 100 tests.
Great.
And you can easily combine multiple properties. Let's say you have 3 Props (p1,p2,p3) the first two of which are correct and the last is not:
scala> (p1 && p2).check
+ OK, passed 100 tests.
scala> (p1 && p3).check
! Falsified after 1 passed tests.
> ARG_0: T(8,true)
scala> (p1 || p3).check
+ OK, passed 100 tests.
Now, if you're like me, and you really want to know what's going on, note that it is possible to "collect" the data that is generated for the tests.
For example...
scala> val p = Prop.forAll((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a })
p: org.scalacheck.Prop = Prop
This is essentially the same thing, but notice the insertion of the call to "Prop.collect".
Now when we check it, we get this...
scala> p.check
+ OK, passed 100 tests.
> Collected test data:
4% (0,0)
1% (-22,38)
1% (4,13)
1% (50,30)
1% (-1,44)
1% (4,28)
1% (12,-16)
1% (-1,-16)
1% (-11,2147483647)
1% (72,-17)
1% (8,-1)
1% (-3,-77)
1% (-46,-19)
1% (-23,-1)
1% (-26,-22)
1% (-21,46)
1% (-48,48)
1% (1,-13)
1% (18,1)
1% (19,-1)
1% (34,-3)
1% (-1,-29)
1% (-58,-63)
1% (1,15)
1% (4,11)
1% (17,-22)
1% (-8,-8)
1% (3,-15)
1% (-9,-36)
1% (-92,78)
1% (28,-42)
1% (-18,0)
1% (3,1)
1% (4,2147483647)
1% (-2147483648,10)
1% (37,-2147483648)
1% (8,-2147483648)
1% (21,0)
1% (1,39)
1% (17,88)
1% (0,-48)
1% (38,-44)
1% (0,6)
1% (2147483647,29)
1% (-1,-24)
1% (-28,11)
1% (-4,46)
1% (28,-63)
1% (35,8)
1% (9,12)
1% (18,29)
1% (8,-18)
1% (-3,-1)
1% (0,14)
1% (47,1)
1% (4,3)
1% (-84,-7)
1% (13,31)
1% (-60,29)
1% (-41,-43)
1% (-75,67)
1% (-3,2)
1% (-73,-57)
1% (-10,-13)
1% (15,-3)
1% (2147483647,-25)
1% (8,0)
1% (-79,76)
1% (48,3)
1% (-5,9)
1% (4,0)
1% (-2147483648,8)
1% (43,-25)
1% (-46,54)
1% (-9,-2147483648)
1% (52,33)
1% (-10,4)
1% (36,-20)
1% (1,0)
1% (30,1)
1% (-2147483648,22)
1% (-27,-12)
1% (0,1)
1% (-58,2147483647)
1% (38,-27)
1% (-38,19)
1% (23,38)
1% (11,-15)
1% (-2147483648,17)
1% (-17,51)
1% (-36,-59)
1% (1,14)
1% (-3,-11)
1% (-31,29)
1% (-60,-44)
1% (2147483647,1)
1% (4,16)
So, we can see what's at work behind ScalaCheck's default generator (Gen) for "Int". A lot of random numbers there. Edge cases, positive and negative, zero, etc. Thanks, ScalaCheck! You just saved me a lot of test writing!
Let's look at a couple of things you can do with "Gen".
First, there's a "choose" method to choose amongst options. Say, we only wanted to check our property with Ints between 1 and 100.
scala> Prop.forAll(Gen.choose(1,100),Gen.choose(1,100))((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a }).check
+ OK, passed 100 tests.
> Collected test data:
1% (32,48)
1% (80,30)
1% (26,31)
1% (15,91)
1% (39,3)
1% (12,68)
1% (93,71)
1% (8,22)
1% (44,58)
1% (4,73)
1% (76,96)
1% (30,98)
1% (55,100)
1% (76,28)
1% (93,13)
1% (46,47)
1% (78,56)
1% (77,30)
1% (69,17)
1% (29,81)
1% (11,39)
1% (40,3)
1% (17,56)
1% (2,81)
1% (31,2)
1% (42,7)
1% (47,22)
1% (48,17)
1% (79,33)
1% (48,40)
1% (52,83)
1% (95,50)
1% (63,30)
1% (21,96)
1% (29,30)
1% (72,33)
1% (70,56)
1% (24,79)
1% (68,16)
1% (89,7)
1% (82,21)
1% (13,35)
1% (73,88)
1% (99,25)
1% (79,8)
1% (52,84)
1% (92,32)
1% (90,77)
1% (88,60)
1% (51,4)
1% (32,93)
1% (62,90)
1% (9,14)
1% (40,36)
1% (78,80)
1% (84,5)
1% (31,63)
1% (7,14)
1% (48,88)
1% (92,21)
1% (31,52)
1% (58,2)
1% (69,82)
1% (84,7)
1% (19,69)
1% (48,94)
1% (50,16)
1% (59,52)
1% (63,63)
1% (55,40)
1% (50,39)
1% (30,26)
1% (15,61)
1% (85,30)
1% (93,90)
1% (19,49)
1% (5,61)
1% (28,56)
1% (41,60)
1% (88,82)
1% (40,62)
1% (96,48)
1% (85,96)
1% (88,7)
1% (1,91)
1% (14,82)
1% (1,56)
1% (24,98)
1% (86,41)
1% (77,65)
1% (4,38)
1% (78,77)
1% (4,62)
1% (84,36)
1% (90,33)
1% (12,67)
1% (45,45)
1% (54,75)
1% (38,94)
1% (14,91)
Or, we can see that the range is inclusive.
Or, we can define a Gen which is based on conditional statements using "suchThat".
val smallOdds = Gen.choose(1,100) suchThat (_ % 2 == 1)
smallOdds: org.scalacheck.Gen[Int] = Gen()
And use that for both our Ints:
scala> Prop.forAll(smallOdds,smallOdds)((a:Int,b:Int) => Prop.collect(a,b) { a+b==b+a }).check
+ OK, passed 100 tests.
> Collected test data:
2% (19,75)
2% (53,55)
1% (35,47)
1% (15,75)
1% (13,41)
1% (51,99)
1% (39,23)
1% (97,35)
1% (47,99)
1% (79,87)
1% (95,55)
1% (45,49)
1% (3,61)
1% (73,93)
1% (13,5)
1% (45,45)
1% (31,51)
1% (61,9)
1% (55,55)
1% (25,47)
1% (99,21)
1% (65,29)
1% (47,49)
1% (47,89)
1% (95,39)
1% (43,73)
1% (39,49)
1% (41,17)
1% (89,25)
1% (25,39)
1% (37,27)
1% (17,49)
1% (77,37)
1% (11,9)
1% (15,13)
1% (37,47)
1% (93,77)
1% (5,75)
1% (19,87)
1% (35,39)
1% (3,21)
1% (65,53)
1% (73,83)
1% (39,63)
1% (31,53)
1% (69,31)
1% (99,65)
1% (9,97)
1% (55,57)
1% (83,71)
1% (41,35)
1% (59,69)
1% (3,55)
1% (85,9)
1% (87,71)
1% (33,33)
1% (9,83)
1% (59,55)
1% (19,51)
1% (49,45)
1% (95,61)
1% (45,21)
1% (57,61)
1% (87,89)
1% (35,71)
1% (85,89)
1% (61,91)
1% (47,5)
1% (3,69)
1% (3,17)
1% (39,85)
1% (95,89)
1% (87,11)
1% (85,49)
1% (55,79)
1% (7,97)
1% (93,29)
1% (37,61)
1% (51,19)
1% (67,53)
1% (71,9)
1% (57,11)
1% (41,85)
1% (31,27)
1% (31,57)
1% (87,49)
1% (69,53)
1% (59,77)
1% (23,87)
1% (29,21)
1% (11,59)
1% (69,69)
1% (77,79)
1% (85,65)
1% (31,31)
1% (97,15)
1% (11,15)
1% (75,85)
Let's say you wanted to generate data for an arbitrary case class. ScalaCheck comes with an "Arbitrary" class to do just that:
scala> case class T(a:Int, b:Boolean)
defined class T
scala>import org.scalacheck.Arbitrary
import org.scalacheck.Arbitrary
scala> val genT = for { a <- Gen.choose(1,10); b <- Arbitrary.arbitrary[Boolean] } yield T(a,b)
genT: org.scalacheck.Gen[T] = Gen()
So, our class should be comprised of an Int value 1 to 10 (inclusive) and a boolean.
scala> Prop.forAll(genT)((t:T) => Prop.collect(t) { t.a >= 1 && t.a <= 10 && (t.b == true || t.b == false) }).check
+ OK, passed 100 tests.
> Collected test data:
9% T(3,true)
8% T(10,true)
7% T(6,false)
7% T(2,true)
6% T(4,false)
6% T(7,false)
6% T(2,false)
6% T(1,true)
6% T(4,true)
5% T(5,false)
5% T(8,true)
5% T(8,false)
5% T(3,false)
4% T(5,true)
4% T(7,true)
3% T(10,false)
3% T(1,false)
3% T(9,false)
1% T(9,true)
1% T(6,true)
Lovely.
Another interesting feature is the ability to classify values that are chosen by the generator. For example...
scala> Prop.forAll(genT)(t => Prop.classify({ t.b==true }, "b is true")(p)).check
+ OK, passed 100 tests.
> Collected test data:
60% b is true
Note: "p" is a property which holds in this case.
Labels:
Let's say you have three properties you are testing, and two of the three are correct...
scala> val p1 = Prop.forAll(genT)((t:T) => t.a >= 1)
p1: org.scalacheck.Prop = Prop
scala> p1.check
+ OK, passed 100 tests.
scala> val p2 = Prop.forAll(genT)((t:T) => t.a <= 10)
p2: org.scalacheck.Prop = Prop
scala> p2.check
+ OK, passed 100 tests.
scala> val p3 = Prop.forAll(genT)((t:T) => t.b == false)
p3: org.scalacheck.Prop = Prop
scala> p3.check
! Falsified after 2 passed tests.
> ARG_0: T(1,true)
Run in combination, it's impossible to know which property failed:
scala> (p1 && p2 && p3).check
! Falsified after 0 passed tests.
> ARG_0: T(2,true)
So, you can label the individual properties using a special ":|" operator:
scala> (p1 :| "a greater than or equal to 1" && p2 :| "a less than or equal to 10" && p3 :| "b always false").check
! Falsified after 0 passed tests.
> Labels of failing property:
b always false
> ARG_0: T(8,true)
Lastly, another cool feature which is very "scala-ish": implicit generator definitions. Basically, ScalaCheck uses implicit definitions to decide which generator to use. By default, there are implicit generators for basic types (Int, String, etc., as seen above where no generator definition was necessary for Ints). However, here's how implicit def works for our case class (T):
scala> implicit def arbT: Arbitrary[T] = Arbitrary(genT)
arbT: org.scalacheck.Arbitrary[T]
scala> Prop.forAll((t:T) => t.a >= 1).check
+ OK, passed 100 tests.
Don't blink or you'll miss the magic!
This is all gone into in much more detail in the ScalaCheck User Guide, but I hope this overview will prove helpful to some.
No comments:
Post a Comment