What Can We Learn From Software Engineers? Part 4 Agile Development

What Can We Learn From Software Engineers? Part 4 Agile Development

Methodology: By what method will we construct software? Process Tools Control 2

Every method must address these issues at some point. Requirements Architecture & design Coding Unit testing Integration testing User acceptance The question is, when are they addressed? And by whom? 3

The waterfall method is the traditional method. Customer fully responsible for rigid, detailed requirements Developer responsible for exact fulfillment Big, slow loop around process Silo'ed responsibilities Little or no feedback Notorious failures Popular with large organizations Clearly defined blame 4

The agile method is a modern alternative. Start with broad requirements Small, fast iterations: sprints Demonstrable results: Show and Tell Fast or immediate feedback Incremental evolution of everything: requirements, design, code, docs,... Best in small teams Nerve-racking for project managers 5

Agile development uses feedback to make constant adjustments in a highly collaborative environment. - Subramaniam and Hunt 6

Waterfall vs. Agile Well-defined but inflexible goals versus incremental refinement Customer/developer split versus teamwork Big bang ending versus incremental products Blame versus engagement & cooperation 7

Sprint The agile method uses some special concepts. MVP = Minimum Viable Product Stubroutines and mock objects Unit testing Continuous testing Continuous integration Continuous releasing Open development meaning open to client 8

First-cut requirements Create a program that will load our data for a given date, fit the model, and summarize/evaluate the model's quality. 9

First sprint: Get something running Goal: Simple load / fit / print with stubroutines. 10

First sprint: Get something running app = function() { } data = loaddata() model = fitmodel(data) summarizemodel(model) It runs. Try it! What's the customer's reaction? 11

Second sprint: Add some output Load dummy data Fit a model... any model. Print a model summary What do you think of that output? 12

Second sprint: Add some output app = function() { data = loaddata() model = fitmodel(data) summarizemodel(model) }... summarizemodel = function(model) { # TODO: Create customized summary print(summary(model)) } 13

Third sprint: Better output Incorporate user feedback: Rework output to be more useful and readable. Result: Yes! That's much better! Can we improve the model? 14

Third sprint: Better output summarizemodel = function(model) { cat("*** Model Summary ***\n\n") cat("f statistic:", summary(model)$fstatistic, "\n") cat("adj'ed R squared:", summary(model)$adj.r.squared, "\n") print(model) oldpar = par(no.readonly=true) par(mfrow=c(2,2)) plot(model) par(oldpar) } # TODO: Write summary to report file, too 15

Third sprint: Better output Incorporate user feedback: Rework output to be more useful and readable. Result: Yes! That's much better! Can we improve the model? 16

Fourth sprint: Better model Address user's concern: Graduate to more sophisticated model. 17

Fourth sprint: Better model app = function() { } data = loaddata() model = fitmodel(data) summarizemodel(model)... fitmodel = function(appdata) { } lm(sepal.width ~ Petal.Width + Species, data=appdata) 18

Fourth sprint: Better model Address user's concern: Graduate to more sophisticated model. Result: This is way cool! First, can you improve the summary? Second, can we get a copy? 19

Fifth sprint: Even better output Address user's request: Can you improve the output further? 20

Fifth sprint: Even better output summarizemodel = function(model) { cat("*** Model Definition ***\n\n") print(formula(model), showenv=false) cat("\n") fstat = summary(model)$fstatistic pval = pf(fstat[1], fstat[2], fstat[3], lower.tail=false) cat("*** Model Statistics ***\n\n") cat("f-statistic:", fstat[[1]], "on", fstat[[2]], "and", fstat[[3]], "DF, ") cat("p-value:", pval, "\n") cat("adj'ed R-squared:", summary(model)$adj.r.squared, "\n\n") cat("*** Coefficients ***\n\n") print(coef(model)) cat("\n") cat("*** Confidence Intervals ***\n\n") print(confint(model)) oldpar = par(no.readonly=true) par(mfrow=c(2,2)) plot(model) par(oldpar) } # TODO: Write summary to report file, too 21

Fifth sprint: Even better output Address user's request: Can we improve the output further? Reaction: Great! Now we see different data? 22

Real-life case study Initial discussions and blue sky First-cut requirements First sprint Skeleton app Client likes skeleton, lukewarm on interface Second sprint Dummy data, MVP output Client excited by usable output, lukewarm on interface 23

Real-life case study (con't) Third sprint More output, better formatting Client likes new reports, lukewarm on interface Fourth sprint Completely rework interface Client much happier Fifth sprint Additional models and reports Client very engaged, begins using MVP 24

Real-life case study (con't) Sixth sprint Draft user docs, a new model Client using MVP, giving feedback and direction Seventh sprint Yet more models and reports Client deciding on final features Seventh sprint Real, multiple datasets (finally!) Final sprints Final features, final docs Client very pleased 25

Some tips for co-existing with project managers Inject waterfall-like elements into project, but with wiggle room Broad requirements, loose plan, flexible schedule Agree up front that requirements, plan, and schedule will evolve Put revision points into plan and schedule Incorporate Show and Tell time into schedule 26