Thursday, September 13, 2012

RESTful Web Services Testing, Q&A

Some background:
I was vaguely pointed at one call in an API, via a 2-page "tutorial" that uses CURL examples. Told "Test this some more." by the guy who'd been doing some amount (none?) of hand "success path" testing via CURL. This has since morphed into "regression testing things, all 12 calls", "we have a build API as well", and "there's this hot new feature for a vendor conference in a couple weeks ..."
There was more, but you get the idea.  There were so more specific "requirements" for the RESTful unit testing environment.
1) Get "smoke test" coverage vs. all the calls 
A sequence of CURL requests to exercise a server can be viewed as "testing".  It's piss-poor at best.  Indeed, it's often misleading because of the complexity of the technology stack.

In addition to the app, you're also testing Apache (or whatever server they're using) plus the framework, plus the firewall, plus caching and any other components of the server's technology stack.

However, it does get you started ASAP.
2) expand / parameterize that 
CURL isn't the best choice.  You wind up writing shell scripts.  It gets ugly before long.

Python is better for this.

Selenium may also work.  Oh wait.  Selenium is written in Python.
3) build out to response correctness & error codes 
Proper design for testability makes this easy.

However.  When you've be tossed a "finished" RESTful web service that you're supposed to be testing, you have to struggle with expected vs. actual.

It's not trivial because the responses may have legitimate variances: date-time stamps, changing security tokens or nonces, sequence numbers that vary.

Essentially, you can't just use the OS DIFF program to compare actual CURL responses with expected CURL responses.

You're going to have to parse the response, pick out appropriate fields to check and write proper unittest assertions around those fields.
4) layer in at least that much testing for the new, new feature breathlessly happening RIGHT NOW.
Without a proper design for testability, this can be painful.

If you're using a good unit test framework, it shouldn't be impossible.  Your framework must be able to start the target RESTful web service for a TestCase, exercise the TestCase, and then shutdown the target RESTful web service when the test has completed.

Now, you're just writing unittest TestCase instances for the new feature breathlessly happening RIGHT NOW.  That should be manageable.
...tool things I've found so far... [list elided
All crap, more or less.  They're REST client libraries, not testing tools.

You need a proper unit testing framework with TestCase and TestSuite classes and a TestRunner.  The tools you identified aren't testing frameworks, they're lower level REST client and client library.  CURL, by itself, isn't really very good for robust testing unless you embed CURL in some test framework.
For defining interfaces (2), I have found these... [list elided]
API's in a typical RESTful environment have little or no formal definition, other than Engrish.  WSDL is for Java/XML/SOAP.  It's not used much for (simpler) REST.  For the most part, REST API definitions (i.e., via JSON or whatever) are mostly experimental.  Not standardized.  Not to be trusted.

The issue is one of parallel maintenance.  The idea is that a REST frameworks can operate without too much additional JSON or XML folderol; just the code should be sufficient.

If there's no WSDL (because it's just REST) then there's no formal definition of anything.  Sorry.
I (perhaps foolishly) figured there's be some standard way to consume the standard format definition of an API, to generate test cases and stubbing at least. Maybe even a standard set of verifications, like error codes. So I went a-googling for 1) a standard / conventional way to spec these APIs, 2) a standard / conventional tool or maybe tools @one per stack, and 3) a standard / conventional way to generate tests or test scaffolding in these tools, consuming the standard / conventional API spec. So far, not so much.
"So far, not so much" is the state of the art.  You have correctly understood what's available.

REST -- in spite of it's trivial simplicity and strict adherence to HTTP -- is a rather open world.  It's also pretty simple.  Fancy tools don't help much.

Why not?

Because decent programming languages already do REST; tools don't add significant value.  In the case of Python, there are relatively few tools (Selenium is the big deal, and it's for browser testing) because there's no real marketplace niche for them.  In general, simple Python using httplib (or Python 3 http.client) can test the living shit out of RESTful API better than CURL/DIFF with no ugly shell-script coding.  Only polite, civilized Python coding.

Tuesday, September 11, 2012

RESTful Web Service Testing


Or they don't have effective lower-level unit tests.







TestCase Overview



When working with SQL, In-memory databases are essential for this.  SQLite (Python) or (Java) can be life-savers because they're fast and flexible.


Mocking The Server




import http.server
import os
from the_application import some_application_feature
class AppWrapper( http.server.BaseHTTPRequestHandler ):
    def do_GET( self ):
        # Parse the URL

        id= url.split("/")[-1]

        # Invoke the real application's method for GET on this URL.
        body= some_application_feature( id )
        # Respond appropriately
        self.send_response( 200, body )
    ... etc ...

# Database setup before starting the service.
# Filesystem setup before starting the service.
# Other web service proxy processes must be started, too.
with open("","w") as pid_file:
    print( os.getpid(), file=pid_file )
httpd = http.server.HTTPServer("localhost:8000", AppWrapper)
    # Cleanup other web services.

Here's a shutdown script.

import os, signal
with open("") as pid_file:
    pid= int( )
os.kill( pid, signal.CTRL_C_EVENT )



Plus,  you can always fall further back to a CGI-like unit testing capability where "body= some_application_feature( id )" becomes a Yes it's inefficient.  We're just testing.


Tuesday, August 28, 2012

Password Encryption -- Short Answer: Don't.

First, read this.    Why passwords have never been weaker—and crackers have never been stronger.

There are numerous important lessons in this article.

One of the small lessons is that changing your password every sixty or ninety days is farcical.  The rainbow table algorithms can crack a badly-done password in minutes.  Every 60 days, the cracker has to spend a few minutes breaking your new password.  Why bother changing it?  It only annoys the haxorz; they'll be using your account within a few minutes.  However.  That practice is now so ingrained that it's difficult to dislodge from the heads of security consultants.

The big lesson, however, is profound.

Work Experience

Recently, I got a request from a developer on how to encrypt a password.  We have a Python back-end and the developer was asking which crypto package to download and how to install it.

"Crypto?" I asked.  "Why do we need crypto?"

"To encrypt passwords," they replied.

I spat coffee on my monitor.  I felt like hitting Caps Lock in the chat window so I could respond like this: "NEVER ENCRYPT A PASSWORD, YOU DOLT."

I didn't, but I felt like it.

Much Confusion

The conversation took hours.  Chat can be slow that way.  Also, I can be slow because I need to understand what's going on before I reply.  I'm a slow thinker.  But the developer also needed to try stuff and provide concrete code examples, which takes time.

At the time, I knew that passwords must be hashed with salt.  I hadn't read the Ars Technica article cited above, so I didn't know why computationally intensive hash algorithms are best for this.

We had to discuss hash algorithms.

We had to discuss algorithms for generating unique salt.

We had to discuss random number generators and how to use an entropy source for a seed.

We had to discuss in some depth, since the algorithms in section 3.2.2. show some best practices in creating hash summaries of usernames, passwords, and realms.

All of this was, of course, side topics before we got to the heart of the matter.

What's Been Going On

After several hours, my "why" questions started revealing things.  The specific user story, for example, was slow to surface.


Partly because I didn't demand it early enough.

But also, many technology folks will conceive of a "solution" and pursue that technical concept no matter how difficult or bizarre.  In some cases, the concept doesn't really solve the problem.

I call this the "Rat Holes of Lost Time" phenomena: we chase some concept through numerous little rat-holes before we realize there's a lot of activity but no tangible progress.  There's a perceptual narrowing that occurs when we focus on the technology.  Often, we're not actually solving the problem.
IT people leap past the problem into the solution as naturally as they breathe. It's a hard habit to break.
It turned out that they were creating some additional RESTful web services.  They knew that the RESTful requests needed proper authentication.  But, they were vague on the details of how to secure the new RESTful services.

So they were chasing down their concept: encrypt a password and provide this encrypted password with each request.  They were half right, here.  A secure "token" is required.  But an encrypted password is a terrible token.

Use The Framework, Luke

What's most disturbing about this is the developer's blind spot.

For some reason, the existence of other web services didn't enter into this developer's head.  Why didn't they read the code for the services created on earlier sprints?

We're using Django.  We already have a RESTful web services framework with a complete (and high quality) security implementation.

Nothing more is required.  Use the RESTful authentication already part of Django.

In most cases, HTTPS is used to encrypt at the socket layer.  This means that Basic Authentication is all that's required.  This is a huge simplification, since all the RESTful frameworks already offer this.

The Django Rest Framework has a nice authentication module.

When using Piston, it's easy to work with their Authentication handler.

It's possible to make RESTful requests with Digest Authentication, if SSL is not being used.  For example, Akoha handles this.  It's easy to extend a framework to add Digest in addition to Basic authentication.

For other customers, I created an authentication handler between Piston and ForgeRock OpenAM so that OpenAM tokens were used with each RESTful request.  (This requires some care to create a solution that is testable.)

Bottom Lines

Don't encrypt passwords.  Ever.

Don't write your own hash and salt algorithm.  Use a framework that offers this to you.

Read the Ars Technica article before doing anything password-related.

Tuesday, August 21, 2012

Performance "Tuning": running in 1/100th the time

For the 757 Python Meetup group, someone proposed looking at some Python code they had which was slow.  The code implemented a variation on the Elo chess rating system.  It applied the ratings to other sports, and used the points scored as well as basic win/lose/tie to work out a ranking for sports teams.  Very clever stuff.


It was described as horribly slow. Since it was only 400 lines of code, it was a great subject for review in a Python meetup.  I would be able to show some tweaks and performance tips.

Hypothetical Step 1:  Profile

My first thought was to run the profiler and see what popped up as a possible root cause of slowness.

Generally, there are three kinds of performance problems.

  • Wrong data structure.  Or wrong algorithm.  (They're simply opposite sides of the same coin.)  This generally leads to dramatic, earth-shattering improvements.
  • Piss-Poor Programming Practices.  This kind of fine-tuning yields minuscule improvements.  In some cases, no measurable change at all.
  • Bugs.  Every time I've been asked to improve "working" code, I find that it has bugs.  Every time.  These can exacerbate performance problems.  Interestingly, most of these are readily apparent during an initial survey:  i.e., while simply reading the code to see how it works.  Trying to create unit tests for the purposes of refactoring often reveals additional bugs.
I didn't know what I was going to see in this 400-line program.

Hypothetically, profiling will help show what kind of problems we have.

Prior to profiling, of course, we need to run the darn thing.  Thankfully, they provided some typical input files.  For example, 1979 High School Football season results.  159 lines of patiently gathered teams and scores.

It Didn't Run

When we tried to run it with the profiler, we found that we had a bug.  Tracking down the bug, revealed the essential performance problem, also.

The core problem was a failure to make use of Python data structures.  

This manifested itself in a number of really bad design errors.  Plus, it presented as the actual, show-stopping, serious bug.

In order to work out team rankings, the program kept a list of teams.

I'll repeat that for folks who are skimming.
Not a dict mapping from team name to team details.  But a simple list.  Locating a team in the list meant iterating through the list, looking for the matching team.  Really.

for t in team_list:
    if t['name'] == target_name:
        process( t )

This kind of thing was repeated in more than one place.

And one of those repeats had the bug in it.

What we have here is "wrong data structure".  Replacing a list with a dict will have earth-shattering impact on performance.

The Bug

The bug, BTW, was a lookup loop which had the additional requirement of adding missing teams.  It tried to use the for-else structure.  This was the intended code (not the actual code).

for t in team_list:
    if t['name'] == target_name:
    t['name']= init_new_team(target_name)
This is a first-class bit of Python.  An else clause on a for statement is a real feature of the language.  However, it's obscure enough that it's easy to get wrong.

However, it's also unique to Python, and the kind of thing that can lead to confusion.  I discourage it's use.  

Test-Driven Reverse Engineering

We're doing TDRE on a little piece of recreational programming.  This means that we need to fabricate unit tests for code that is purported to work.  Some folks like to call this "exploratory testing" (a phrase which is contradictory, and consequently meaningless.)  We're "exploring".  We're not "testing".

Once the core bug is fixed, we can run sample data through the application to get "big picture" results.  We can extract bits from the sample data and exercise various functions in isolation to determine what they actually do now, bugs included.

Since this is so simple--and we're going to review it during a 2-hour meet up--and there's nothing billable going on--we can get by with a few really simple unit tests.  We'll run the darn thing once to create some expected output.  We save the output and use that single output log as the expected results from each refactoring step.

More complex applications require more unit tests.  For a billable TDRE effort last year, I had to create 122 unit tests to successfully refactor a program of over 6,000 lines of code.  It took several weeks.

Real Step 1: Fix The Data Structure

Before profiling (but after running to capture some output) we have to fix the essential data structure.  Repeated scanning of a list is never going to perform well.
The whole point of the study of "Data Structures" is to prevent (or optimize) search.
In this case, we can prevent linear search of a list by using a dictionary.  That, clearly, is job one.

It was a pervasive rewrite.   Essentially, everything in this little program included a goofy loop to lookup a team in the list.  The Elo algorithm itself, which is already O(n2), is dragged down by using the linear search for the opposing team four more times, making it O(n3).

Cyclomatic Complexity

One of the big "issues" is the use of if statements throughout the scoring algorithm.  An if statement creates Cyclomatic Complexity and can lead to performance problems.  Generally, if statements should be avoided.

This algorithm applies some normalization factors to reconcile scoring with win/loss numbers in different sports.  Basketball, specifically, involves generally high scores.  Since there are 2-point and 3-point scoring opportunities, a factor is used to normalize the points into "goals".  Football, similarly, has numerous scoring opportunities with values of 1, 2, 3 and 6 points; the scores here are also normalized.

This normalization was done with an if statement that was evaluated deep inside the Elo algorithm.  Repeatedly. Evaluated.

The two functions that handled the normalizations, plus the normalization factors, are ideal candidates for OO design.  There's a clear hierarchy of classes here.  A superclass handles most sports, and two polymorphic subclasses handle football and basketball normalization.

The if statement is now "pushed up" to the very beginning of the program where an instance of the sports normalization object is created.  This object's methods are then used by the Elo algorithm to normalize scores.

Icing on the Cake

Once we've fixed the bug and replaced a list with a dict, everything else is merely icing.
Some other OO changes.
  1. The "Team" information should not be a flat, anonymous dictionary.  It should be a proper class definition with proper attributes.  There aren't many methods, so it's easy to create. 
  2. The "Game" information is read by csv.DictReader.  However, it should not remain a simple, anonymous dict.  As with a Team, a simple class can be created to handle Game.
  3. The overall structure of the application needs to be broken into two sections.  The command-line interface parses options, opens files, and generally gets everything set up.  The actual ranking algorithm should be a function that is given an open file-like object plus the Sport object for normalization.  This allows the ranking algorithm to be reused in other contexts than the command-line (i.e. a web service).
A more subtle OO design point is the question of "mutability".  A Team in this application is little more than a name.   There are also numerous "stateful" values that are part of the Elo algorithm.   A Game, similarly, is an immutable pair of teams and scores.  However, it has some mutable values that are part of the Elo algorithm.  

Really, we have immutable Team and GameHistory objects, plus a few values that are used as part of the Elo calculation.  I'm a big fan of disentangling these mutable and immutable objects from each other.  

I suspect that the Elo algorithm doesn't really need to update the "state" of an object.  I suspect that it actually creates (and discards) a number of immutable candidate ranking objects.  The iteration that leads to convergence might be a matter of object creation rather than update.  I didn't make this change, since it required real work and we were out of time.

Bottom Line

The more times I do TDRE to improve performance, the more I realize that it's all about bugs and data structures.

This recreational application took 45-60 seconds to process one year's record of games for a given league.  It now takes less than 0.2 seconds to do the same volume of work.  Two test cases involving a complete run of 159 records runs in 0.411 seconds.  That's 1/100th the time simply from switching data structures.

The idea of "tweaking" a working program to improve performance is generally misleading.  It might happen, but the impact is minuscule at best.  

Here's the checklist for getting 100:1 improvements.
  • Remove searches.  
  • Remove deeply-nested if statements.  
Generally, reduce Cyclomatic Complexity.

Tuesday, August 7, 2012

How Expensive is a "Waterfall" Project Plan?

It's impossible to step into the same river twice; other waters are flowing toward the sea.  It's impossible to do "head-to-head" project comparisons.  You can't have the same people doing the same thing with only one constraint changed.  You can try to resort to having similar people doing the exact same thing.

It's impossible to provide trivial quantification of the costs of a waterfall approach.  However, there's a lot of places where "up front" engineering adds cost and risk.  A waterfall plan adds friction, zero-value administrative costs and (sometimes) extra work.

For software development that doesn't involve "toy" projects, but apply to the magical "real world" projects, it's difficult to justify building the same "real world" thing twice.  It's hard to create a detailed and purely economic analysis of this question.  Therefore, we have to resort to hand-waving and speculation.

[I put "real world" in quotes, because the "real world" is a kind of dog-whistle phrase that only project managers understand.  It appears to mean "projects that fit my preconceived notions" or "projects that can't be done any other way because of political considerations" or "projects constrained by a customer to have a farcical structure".  Another variant on this dog-whistle phrase is "the realities of running a business."  There's something in their real world that they (a) can't define and (b) prevents any change.]

A few months back, I saw a presentation that was intended to drive architecture, cost and schedule conversations.  The authors were happy.  I was left relatively mystified.

There are three relevant issues that don't fit well into a prezi presentation.
  1. Open Technology questions.  If it's new software development, there are unanswered software API, performance, quality, fitness-for-purpose questions.  If it's a clone or a clean-room reverse engineering, then there may be few questions.  Otherwise, there must be unknowns.
  2. Open User Story questions.  If it's new software development, then the user stories are not complete, consistent and optimized.  There must be something missing or extra.
  3. Open User Experience questions.  I'm not a UX person.  Tinkering with an Arduino has taught me that even in the most prosaic, simplistic project there is a UX element.  
It's easy to trivialize this as "details" or "supporting information" omitted from the presentation.  Actually, it's considerably more interesting that simply elaborating details.  Indeed, the missing stuff is often more important that the elements of the presentation than were provided.

The Standard Gaps

While a presentation can show some elements of UX, it's not interactive.  It cannot provide any useful depth on the UX.  Failure to do UX exploration (with real interaction) is crazy.  Making assumptions about UX is more likely to create cost and risk.

User Stories can be kicked around in a presentation.  However.  Once the highest-priority user stories are built, and lessons learned about the users, the problem and the emerging solution, then the other user stories can (and should) change based on those lessons learned.  Assuming a list of user stories and then committing to a specific schedule enshrines the costs in a bad plan.  And we all know that the plan (and the dates) are sacred once written down.

Software does not exist in a vacuum.

I'll repeat that, since it's a hidden assumption behind the waterfall.

Software does not exist in a vacuum.

You have a Heisenberg problem when you start building software and releasing it incrementally.  The presence of the new software, even in a first-release, beta-test, barely-working mode changes the ecosystem in which the software exists.  Other applications can (and will) be modified to adjust to the new software.  Requirements can (and should) change.

A waterfall project plan exists to prevent and stifle change.  A long list of tasks and dates exists to assure that people perform those tasks on those dates.  This is done irrespective of value created and lessons learned.

Technology changes.  Our understanding of the technology changes.  One wrong assumption about the technology invalidates the deliverables for a sprint.  In a project with multiple people, multiple assumptions and multiple sprints, this effect is cumulative.  Every assumption by every person for every technology is subject to (considerable) error.

Every technology assumption must be looked at as a needless cost that's being built into the project.

Recently, I've been working on a project with folks that don't know Django very well.  Their assumptions are -- sometimes -- alarming.  Part way through a sprint, I got a raft of technical questions on encrypting passwords.  It's hard to state it strongly enough: never encrypt a password.  What's more useful is this: always hash passwords.  The original password must be unrecoverable.  There are lots of clever hashing techniques.  Some of which are already part of Django.  A significant portion of the sprint (appeared) to be based on reinventing a feature already part of Django.

Do the math: a few wrong assumptions made during project planning are canonized forever as requirements in the project.  With a waterfall project, they're not easy to change.  Project managers are punished for changing the project.  You can't increase the project deadline; that's death.  You can't decrease it, either: you get blamed for sand-bagging the estimates.

Arduino Technology Lessons Learned

After buying several 74HC595 shift registers from Sparkfun, I realized that my assumptions about the interfaces were utterly wrong.  I needed to solder a mix of right-angle header and straight-through headers onto the breakout boards.  I tried twice to order the right mix headers.  It seems so simple.  But assumptions about the technology are often wrong.

This is yet more anecdotal evidence that all assumptions must be treated as simple lies.  Some managers like to treat assumptions as  "possibly true" statements; i.e., these are statements that are unlikely to be false.  This is wrong.  Assumptions always have a very high probability of being false, since they're not based on factual knowledge.

Some managers like to treat assumptions as "boundary conditions":  i.e.,  if the assumption is true, then the project will go according to plan.  Since all of the assumptions will be prove to be incorrect, this seems like simple psychosis.  

[Interestingly, the assumptions-are-boundary folks like to play the "real world" card: "In the real world, we need to make assumptions and go forward."  Since all assumptions will be shown to be incorrect, why make them?  Wouldn't it be more rational to say "we need to explore carefully by addressing high-righ unknowns first"?  Wouldn't it be better to both gather facts and build an early release of the software at the same time?]

Arduino User Story Assumptions

After building a prototype that addressed two of the user stories, it slowly became clear that the third user story didn't actually exist.

Sure, the words were there on paper.  Therefore, there's a user story.


There was nothing to actually do.  

The whole "As a [role], I need [feature] so that I can [benefit]" that was written on day one was based on a failure to understand precisely how much information was trivially available.  The benefit statement was available with a small software change and no separate user story.  And no separate hardware to support that user story.

Exploration and tinkering reduced the scope of the work.  In the real world.

[In the "real world" where waterfall is considered important, exploration is described as unbounded spending of limited resources.  In the real real world, money must be spent; it can either be long hand-wringing meetings or it can be prototype development.]

The user story (written before a prototype existed) was based on a failure to fully understand the UX.  The only way to fully understand the UX is to build it.  Education costs money.

Arduino UX Learnings

Perhaps the most important issue here is UX.

Once upon a time, UX was expensive, difficult and complex.  So difficult that special-purpose prototyping tools were created to make it possible to create a preliminary UX that could be used to confirm UX and user stories.

This UX prototyping effort was real money spent as part of "requirements" or "design"; it created documentation that flowed over the waterfall for subsequent development.

This notion is obsolete.  And has been obsolete for several years.

UX is now so easy to build that it makes much more sense to build two (or three) competing UX's and compare them to see which is actually better.

Indeed, it makes a lot of sense to build one UX and release it; make a real effort at solving the user's problems.  Then build a second UX for A/B testing purposes to see if there's room for improvement.

I'll repeat that for folks who really like the waterfall.

It's now cheaper to actually build two than it is to write detailed requirements for one.

[In the "real world", this is deprecated as "wasting time playing with the UX".  As if written requirements based on nothing more than a whiteboard are more "real" than hands-on experience with the UX.]

You can prove this to yourself by actually observing actual UX developers knocking out pages and screens.  Long "requirements gathering" meetings with prospective users amount to a waste of time and money.  Long "brainstorming" sessions, similarly, are wasteful.  Short, ongoing conversations, a build, a release, and a follow-up review has more value, and costs less.  

Do the math.  Several users and business analysts in several multiple-hour meetings costs how much?

A pair of developers banging out fully-functioning, working UX for a use case costs how much?

A slavish adherence to waterfall development creates "real world" costs.

Thursday, July 12, 2012

Innovation, Arduino and "Tinkering"




  • What's the leading indicator of failure?



There are two central parts of the Arduino philosophy.

  • Tinkering.
  • Interaction Design.







At this point, we've tinkered.  Heavily.





Interaction Design






Enter Arduino










Tuesday, July 10, 2012

Cool stuff I saw at MADExpo

A random list of cool things.



Tuesday, July 3, 2012

Book Deal Fell Apart (sigh)

After spending a couple of years (really) working with a publisher, the deal has gone south.

The problem was—likely—all mine.  The book wasn't really what the publisher wanted.  Perhaps it was close and they thought they could edit it into shape.  And perhaps I wasn't responsive enough to criticism.

I like to think that I was responsive, since I made almost all of the suggested changes.

But the deal fell apart, so my subjective impression isn't too relevant.

What To Do?

I have two essential choices here.
  1. Finish up.  Simply press on with the heavily-edited version.  The path of least resistance.
  2. Roll back the content.  This involves going back to a less heavily edited version and merging in the relevant changes and corrections from the writing process.
Option 2 is on the table because my editor was unhappy with "digressive" material.  The editor had a vision of some kind of narrative arc that exposed Python in one smooth story line.  My attempted presentation was to expose language features and provide supporting details.

Perhaps I'm too deeply invested in the details of computer science.   Or.  Perhaps I'm just a lousy writer.  But I felt that the digressions were of some value because they could fill in some of the gaps I observed while coaching and teaching programmers over the last few decades.

In addition to the editorial challenge, there's technical challenge.  Do I step back from LaTeX?
  • Use LaTeX.  This means that I would have to create the non-PDF version with a LaTeX to HTML translator.  Read Converting LaTeX to HTML in the Modern Age.  See Pandoc.  Or, it means that I don't offer an HTML version (a disservice, I think.)  Also, I need to unwind the publisher's LaTeX style libraries and revert to a plain LaTeX.  Since LaTeX is semantically poor, I need to rework a lot of RST markup.
  • Revert to RST.  While RST tools can make both LaTeX and HTML from a common source, it does mean that some of the fancier LaTeX formatting has to go.  Specifically, I really got to know the algorithm and algorithmic packages.  I hate to give those up.  But maybe I can work something out with Sphinx 1.1.3's various features.
The problem is that three of the four combinations of paths have advantages.
  • Finish up using LaTeX is easiest.  Remove the publisher's document style.  Use HeVeA or Pandoc to make HTML.  Move on.  
  • Finish up but revert to RST.  This means conversion from the publisher's LaTeX to RST followed by some editing to fix the missing RST semantic markup and then some debugging to get the LaTeX and HTML to look good.
  • Rollback the content using LaTeX.  This would be challenging because I would have to merge manually edited publisher-specific LaTeX from the heavily-edited version with the Sphinx-generated LaTeX from the less-heavily edited source.  Things like the publishers' style tags would complicate the merge. 
  • Rollback the content and revert to RST. This means using Pandoc to convert the heavily-edited LaTeX to RST. Then merging the relevant edits into the RST original text.   This actually seems pretty clean, since the heavily-edited RST (converted from LaTeX) would be short.
Perhaps, if I was a better writer, I wouldn't have these problems.

It appears than the solution is MacFarlane's Pandoc.  This can reverse LaTeX to RST, allowing easy side-by-side merging of texts from various sources.  Or.  It can convert LaTeX to HTML, allowing easy work with the heavily edited LaTeX version.

Thursday, June 28, 2012

How to Write Crummy Requirements

Here's an object lesson in bad requirements writing.

Some problems.
What's left?  


Which brings up another important suggestion.
  • Don't listen to the users.



And that leads to another important suggestion.
  • Don't fantasize.




Tuesday, June 26, 2012

Thursday, June 21, 2012

QR Code

I suddenly realized that QR Codes are everywhere.

Except my business cards.

That should allow me to solve that problem and move on.

Tuesday, June 19, 2012

Dereliction of Duty

Recently started looking into Metadata Registries and UDEF and related semantic technology.

The Wikipedia page lists a bunch of relevant Metadata Registry projects and commercial products.  Very nice.  Easy to follow the links and determine features and benefits.


A client has IBM Cognos.  Is there any easy to to see what kind of Metadata features are part of Cognos?

No.  Not really.

I wondered about this marketing gap.  Why doesn't IBM (or Oracle, they're pretty bad at this also) provide a tidy list of features?

  • They're so big (and arrogant) that they don't feel the need to do any marketing?
  • They're so paranoid that they don't want to have their products reduce to a simple bullet list?
  • They're sales people are so good that they don't need a web presence to sell their products?
  • They already have such tight wired-in relationships that they don't need to do any marketing?
Or is it the "growth by acquisition" problem?  Since IBM acquired Cognos, they hesitate to commit to a list of features?

Whatever the reason, it's frankly difficult to include IBM products in an easy-to-understand info-graphic comparing alternative products.

Thursday, June 14, 2012

IBM RAMAC Device: 5 MB

Check out this picture.

Random Reminiscing Follows

When I was in college (1974-1978) 64K of RAM was the size of a refrigerator.

By 1982, 64K of RAM was an Apple ][+ fully tricked out with the 16K expansion card.

I vaguely remember working with a "tower" device that was a 5MB disk drive.  Think Mac Pro case to hold a disk drive.

By 1985 or so, 128K or RAM was a Macintosh and a 5MB disk drive a big desktop console box.  Smaller than a tower.  Irritating because it took up so much real-estate when compared with the Mac itself that was designed to take up an 8 x 11 space (not including keyboard or mouse).

Now 5MB is round-off error.

What's Important?

Has computing gotten that much better?

Old folks (like me) reminisce about running large companies on small computers.  Now, we can't even get our coffee in the morning without the staggering capabilities of a 32GB iPhone.

Old folks, however, are sometimes mired in bad waterfall development methods and don't understand the value of test-driven development.  While the hardware is amazing, the development tools and techniques have improved, also.

Tuesday, June 12, 2012

The Universal Data Element Framework (UDEF)

Okay.  This is seriously cool.

The Universal Data Element Framework (UDEF)  provides a controlled vocabulary that should be used to seed a project's data model.



We're looking at applying UDEF retroactively to an existing schema.  What a pain in the neck!

Step 1.  Parse the table names.  In our case, they're simply CONTIGUOUSSTRINGSOFCHARS, so we have to work out a quick lexicon and use that to break the names into words.  Then we can find the obvious aliases, spelling mistakes and noise words.  'spec', 'quanitity' and 'for' are examples of each.

Step 2.  Look up the various words in the UDEF vocabulary to create candidate matches.   Since each individual word is matched, each table will have multiple candidate matches to seed the analyst's thinking.

Step 3.  Manually pick UDEF standard names or create internal extensions to the standard for the problem domain or enterprise unique features.

Do a similar thing for the column names.  In that case, they're CamelCaseWithSomeACRONYMS.  This is slightly easier to parse, but not much.

Eventually, we have to apply real human business analyst grey matter to locating standard names which might fit with the host of legacy names.

Here's the column name parser.

def property_word_iter( prop_name ):
    """Find words via case changes.
    -   Lower to upper ends a word.
    -   Upper to lower ends a word.  However.
        Sometimes the Upper is an acronym that was all caps.
        A lookahead is required to disambiguate.
    cc_iter= iter(prop_name)
    word=[ next(cc_iter) ]
    for c in cc_iter:
        if c.isdigit():
            yield ''.join(word)
            yield c
            word=[ next(cc_iter) ]
        if word[-1].islower() and c.islower():
        elif word[-1].isupper() and c.isupper():
        elif word[-1].islower() and c.isupper():
            yield ''.join(word)
            c2= next(cc_iter)
            if c2.isupper():
                word= [c, c2]
                word= [c.lower(), c2]
        elif word[-1].isupper() and c.islower():
            c0 = word[-1]
            yield ''.join(word[:-1])
            word= [c0.lower(), c]
            raise Exception( "What? {0!r} {1!r}".format( word[-1], c ) )
    if word:
        yield ''.join(word)

Thursday, June 7, 2012

Stingray Schema-Based File Reader

Just updated the Stingray Reader.  There was an egregious error (and a missing test case).  I fixed the error, but didn't add a test case to cover the problem.

It's simple laziness.  TDD is quite clear on how to tackle this kind of thing.  Write the missing test case (which will now fail).  Then make the code change.

But the code change was so simple.

Tuesday, June 5, 2012

COBOL Rework

See this article: "The COBOL Brain Drain" in ComputerWorld.  This article is very, very well written and repeats a number of common fallacies.

The fallacies lead to expensive, labor-intensive in-house software development and maintenance.  Either there's a lot of effort poking at the COBOL code.  Or there's a lot of extra code to "wrap" the COBOL code so that it's never touched.

"Migrating large-scale systems built in Cobol is costly and risky."  A popular position.  But the risks are actually quite small; possibly non-existent.  The risks of not converting are usually higher than the risk of conversion.

The perception of the COBOL code is that it's filled with decades of tricky, nuanced legacy details that are hard to fully understand.  This is only partially true.

A great deal of the tricky code is simply redundant.  COBOL is often written with copy-and-paste programming and blocks of code are simply repeated.  It's also important to note that some of the code is no longer exercised in the first place.

Mythical Risk

The "risk" comes from the perceived opacity of  the tricky, nuanced legacy details.  It doesn't appear to be clear what they mean.  How can a project be started when the requirements aren't fully understood?

What appears to be the case in reality is is that this tricky code isn't very interesting.  Most COBOL programs don't do much.  They can often be summarized in a few sentences and bullet points.

Detailed analysis (641,000 lines of code, 933 programs) reveals that COBOL programs often contain several commingled feature sets.

  • The actual business rules.  These are often easy to find in the code and can also be articulated by key users.  The real work is usually quite simple.  
  • A bunch of hackarounds.  These are hacks to work around bugs that occur elsewhere in the processing.  Sometimes a hackaround introduces additional problems which require yet more hackarounds.  All of this can be ignored.  
  • Solutions to COBOL data representation issues.  Most of these seem to be "subtype" issues: a flag or indicator is introduced to distinguish between subtypes.  Often, these are extensions.  A field that has a defined range of values ("A", "C" or "D") has a few instances of "*" to indicated another subclass that was introduced with a non-standard code for peculiar historical reasons.
Once we separate the real code from the hackarounds and the representation issues, we find that most COBOL programs are trivial.  About 46% of the lines of code (74% of the distinct programs) involves programs that read one to four files to write a report.  In effect, these programs do a simple "relational join" or query.  These programs have single-sentence summaries.  

The hackaround problem is profound.  When looking at COBOL code, there may be endless copy-and-paste IF-statements to handle some issue.  There may be whole suites of programs designed to work around some issue with a third-party package.  There may be suites of programs to deal with odd product offerings or special customer relationships.  

The remaining 26% of the non-trivial programs follow a normal distribution of 1/4 simple, 1/2 moderately complex, and 1/4 regrettably and stupidly complex.  That final 5% of the programs will also be a whopping 20% of the lines of code.  They are the few big programs that really matter.  

Risk Mitigation

The risk mitigation strategy involves several parts.  
  1. Data Profiling.  The COBOL data may have considerable value.  The processing is what we're trying to rewrite into a modern language.  A profile of every field of every file will reveal that (a) most of the data is usable and (b) the unusable portion of the data isn't very valuable.   
  2. Triage.  We can summarize 80% of the code in simple sentences.  46% of the code is single-sentence summaries.  34% of the code has multiple sentence summaries.  The remaining 20% requires in-depth analysis because the programs are large; they average of 2400 lines of code each.
  3. Test-Driven Reverse Engineering.  Since a 5% of the programs do the real updates, it's important to collect the inputs and outputs of these few programs.  This forms a core test suite.
  4. Agile Methods.  Find the user stories which really matter.  Find the COBOL programs that implement those user stories.
The most important risk reduction strategy is to take an Agile approach.  We need to prioritize based on value creation.  All COBOL programs are not created equal.  Some fraction can simply be ignored.  Another fraction produces relatively low value, and can be converted last.

The second most important risk mitigation is to build conversions from legacy COBOL files to the new, preferred databases.  The data is preserved.  

There's almost no risk in rewriting the 46% of low-complexity COBOL lines of code.  This code is trivial.  Note that some of this code actually has no business value.  Simply ignoring the no-value code also reduces risk.  Since we're using live files to drive testing, we can easily prove that our new programs work.

It's not risky to focus on the 20% of high-value COBOL lines of code.  This contains most (or all) of the processing that the business needs to have preserved.  They can articulate the user stories; it's easy to confirm that the COBOL does what the business needs.  It's easy to use live data to drive the reverse engineering.

The remaining 34% of the code base may actually involve a small amount of overlooked complexity.  There may be a nuance here that really matters.    

This overlooked "nuance" is something that the users didn't articulate, but it showed up in our unit testing.  We couldn't reproduce an expected output because we didn't correctly locate all the processing steps.  It wasn't in our summary of the 80% of moderate-to-low complexity programs.  It wasn't in our detailed analysis of the 20% subset of hyper-complex, large programs.  

We've reduced the risk by reducing the volume of code that needs to be examined under the microscope. We've reduced the risk by using live files for parallel testing.  We've reduced the risk by collecting user stories.

The remaining risks are ordinary project risks, unrelated to COBOL or legacy data.

The Other Great Lie

Another popular fallacy is this:


The value-add is to create a nimble business.  Purging the COBOL has a lot of advantages.

  • It reduces the total number of lines of code.  Reducing costs.  Improving time-to-market for an IT solution to a business problem.
  • It reduces the number of technologies competing for mind-share.  Less thinking about the legacy applications is less time wasted solving problems.
  • It reduces the architectural complexity.  If the architecture is a spaghetti-bowl of interconnections between Web and Legacy COBOL and Desktop, then following the spaghetti-like connections is simply a kind of intellectual friction.
The COBOL does not need to be purged all at once through a magical "big-bang" replacement.  It needs to be phased out.  

Agile techniques need to be applied.  A simple backlog of high-value COBOL-based user stories is the place to start.  The prioritization of these stories needs to then be clustered around the data files. 

Ideally all of the programs which create or update a given file (or related group of files) can be rewritten in a tidy package.  The old files can be used for Test-Driven Reverse Engineering.  Once the programs have been rewritten, the remaining COBOL legacy applications can continue to operate, using a file created by a non-COBOL application.

Each file (and related cluster of programs) is replaced from high-value down to low-value.  Each step creates a more nimble organization.

Thursday, May 31, 2012

The Passive-Aggressive Programmer (again)




But they prefer to passively "fail to agree."


Recently, I received this.







Tuesday, May 29, 2012

Mid-Atlantic Design Expo (MADExpo)


I'm looking forward to this.  I'll be talking about Python.  

  • A Python 3.2 tutorial.  I did a dry run with the help of the 757 Python group.  Made a bunch of changes based on their input.
  • A more in-depth presentation on a good architecture of Database Schema Migration scripts.  You know, the "open heart surgery" of a database where the structure changes in some way that's not a trivial ALTER statement.

Thursday, May 24, 2012

Are engineers valued?

Check out this post: "Why Quit? Because They Have Bigger Monitors".

Brilliant summary of a few small cost things that have a large impact.

How many other signals are being sent?

Jim B. once pointed out that some organizations are entirely run by the accountants.  You can have a good idea.  You can justify the good idea.  You can show ROI for the good idea.  And you can still be shot down in flames with feeble excuses like "it wasn't part of this year's budget, so we won't do it."

Tuesday, May 22, 2012

Delusional Project Managers

Maybe it's the circle-R in the PMP® signature that put me off.

What's important is that the PMP-certified project manager is absolutely married to fixed-price software development.  The idea of "Agile" was unthinkable.  They were very clear that we had enough information to create a fixed price.

It's hard to disagree with someone who's sure that their presentation is a pinnacle of project specification.

Slides 2 and 3 are an overview that lists some broad, vague buzzwords.  ("Asset", "Resource")

Slides 4 to 7 are blurry screen shots of the legacy application.  There must be dozens of pages.  We were shown four.

Slide 8 is a list of project objectives with no priorities, actors or user stories.  "Modify tool navigation to be more intuitive and efficient."  That's the kind of stuff that's supposed to define the scope with enough detail to create a fixed price.

Slide 9 is  another list of objectives.  "Create a simple project and resource KPI dashboard showing real-time status."  I'm not even sure we can define a fixed-price task for figuring out what this might mean.

Slide 10 is a schedule in which all of the dates are already in the past.

We were told -- firmly and finally -- that we were supposed to fabricate a fixed price proposal from this PPT.  We were encouraged to ask follow-up questions.

My first follow-up question should be "Are you crazy?"

My second follow-up question should be "How much time are you willing to put into this effort?  If it's less than 4 hours per day, we can't really help much."

I'm guessing that we're just "column fillers".  There's an incumbent team that has an existing proposal to extend or rewrite; we're merely providing an outside, "independent" quote that can be used for comparison purposes.  The incumbent team probably knows (in detail) the actors and user stories, and has a carefully prioritized feature set.  We don't even know (precisely) what's in the data model.

I would like to avoid writing a "column filler" proposal that will be a million dollars more than they had in mind.

An Agile approach (5 people × 8 hrs × 15 days = 600 labor hours per sprint) was refused.

If they would reveal their budget, we could have a more meaningful conversation on what they can get finished and delivered for that little dribble of money.

I think that PMP best practices are getting in the way of a successful software development effort.

Thursday, May 17, 2012

Flickr, Innovation and Integration


Compelling stuff: "Integration Is The Enemy of Innovation".



Tuesday, May 8, 2012

More of Disruptive Technology Change

There's a cool infographic on technology change in FrugalDad.  See The Great Disruption: The Future of Personal Tech.  It's interesting and informative, but the few predictions it makes are not really disruptive.  You wouldn't see anyone lobbying against the suggested future directions.  They're all good ideas that leverage existing technology.

On the other hand, there's a great graphic that shows how disruptive technology is labeled as illegal.  See Infographic: Why the movie industry is so wrong about SOPA.

Consider just one example.  Digital Movies.  The DVD was so frightening to movie producers (or distributors or theaters or the whole supply chain) that discussion of circumvention of DVD encoding had to be made illegal.  That kind of industry legislative action is evidence that a technology is truly disruptive.

Disruptive change will often lead to fearful rejection and legislative action.

"But wait," you say, "no one tried to make the iPod illegal."  Correct.   The iPod is not the core disruptive change.  Digital music is the disruptive change.  The iPod is just a vehicle.  Apple is making their money by providing a platform for digital content.

If you want to know what the Next Big Thing is, look to the US Congress.  Lobbyists are trying to make some things illegal merely because they're disruptive.

Universal Health Care, as one example, is being fought against.  There are lots of specious and farcical reasons being used to argue against simplifying the insurance mess that has emerged over the last few decades.  If Congress is fighting against it, that means the following:

1.  It's disruptive.  Game Changing.  Terrifying.
2.  The old school companies are spending huge lobbying and campaign budgets to prevent change.  They are unable to adapt to a different set of rules.
3.  Some new school companies stand to be wildly profitable if the change ever gets past the Congressional objections.

For another example, read this brilliant article: How Ma Bell Shelved the Future for 60 Years.  This an example of internal censorship of disruptive technology.  "More precisely, in Bell's imagination, the very knowledge that it was possible to record a conversation would "greatly restrict the use of the telephone," with catastrophic consequences for its business. Businessmen, for instance, the theory supposed, might fear the potential use of a recorded conversation to undo a written contract."

You know it's disruptive when it's actively feared.

Thursday, May 3, 2012

Bravo -- Iteration and Generator Functions

See this:

Nicely done, very thorough presentation on some Python fundamentals.

The nature of iteration and generator functions is easily overlooked.

Thursday, April 26, 2012



The Mid-Atlantic Developer Expo.

In my backyard (more-or-less).

There's going to be this:

and this:

I'm looking forward to, which I'm certainly going to attend.  Many other sessions look like fun, too.

Thursday, April 19, 2012

Should the CIO Know How to Code?

Read this Computerworld posting: Should the CIO know how to code?

The answer is "Yes."

The examples of "well-functioning non-technical CIOs" are people as rare as hen's teeth.  "These are leaders who know what they don't know. They are good at asking the right questions, probing for further insight, and then re-framing the answers in such a way that the business side will understand".

I'm sure there are people like this.  In the last 35 years, I've met very, very few.  Two actually.

Larry and Chuck are the two examples.

Larry knew what he didn't know.  He took the time to actually sit with actual developers and actually watch them work.  It was weird the first time he sat and watched you type.  But without deep knowledge, he couldn't be sure the projects would get done.  So he allocated an hour or more each day to sit with key developers and learn.

Chuck did essentially the kind of thing.  He sat with each developer individually every single day.  He did not have all-hands meetings that lasted hours.  He did not have an "around the table" where everyone spent 20 minutes boring the entire rest of the team with irrelevant details.

Could they code?

Essentially, yes.  They looked at code over a developer's shoulder.  They participated in a form of "pair programming" where they watched code happen.  By themselves they couldn't code much.  As pair programmers, however, they could work with another programmer and get stuff done.

Tuesday, March 27, 2012

Patents vs. Innovation




Thursday, March 22, 2012

Detailed Analysis of Disruptive Technology Change


Don't be afraid to read it again.





What I find endlessly funny are folks who want "formal research" or "studies" that prove that some new idea is actually better than existing ideas.  You can read Stack Overflow and questions looking for studies that prove the value of unit testing or prove the value of a NoSQL database or prove that software is simpler without triggers or stored procedures.

For the moment, these are disruptive ideas.

We know they're disruptive because people keep asking for proof.
