Jack Notarangelo's Application Engineering Blog

Development Managers in Agile

2011-10-30T10:32:00.000-07:00

To understand the role of a development manager in Agile, we must first understand the difference between management and leadership.

The goal of management is to attain perfection in its processes in an effort to maximize productivity. The goal of leadership is to be an agent for change in an effort to maximize organizational sustainability through innovation and adaptability.

Although they are counteracting forces they are both required to be successful. It is as important to get things done efficiently as it is to innovate and adapt.

So how should leadership and management manifest themselves in an Agile-based organization?

From a management perspective, Agile assigns management roles through clear delineation of authority between the Agile roles, i.e. Product Owner, ScrumMaster, and Team.

Leadership is an implied attribute in Agile. Agile consciously acceptances that much of what we believe to be true will prove to be either fundamentally flawed, or, will become flawed because of real-world change. Since change is inevitable, leadership is a universal trait that is necessary for Agile entities to be successful.

A leader is often characterized as a charismatic and/or visionary manager. As I mentioned in my previous blog post, that is because most organizations are autocratic in nature and thus leaders are part the management tier.

Agile is not autocratic. There is no hierarchy. As part of this structure, it is intended that leaders emerge and assert themselves as part of the process of getting things done. This also means that leadership resides in each of the three Agile roles in a quantity that is not assigned or dictated but emerges according to need.

So if Agile groups are not hierarchical and leaders emerge naturally, then what is the role of the development manager?

The job of a development manager in an Agile organization is to promote both leadership and management; to value both within the organization; and to empower both to grow and change organically through the Agile principles. If it sounds like a development manager is more leader than manager - you are right.

I'm sure as you read this, many of you recognize command-and-control (autocracy) as a significant attribute within your workplace. Through your commitment to Agile principles, you are provided the opportunity of implementing the aforementioned organizational model - if not throughout your organization, then as a model for your organization.

Self-Managing Teams, Adaptability and Innovation

2011-10-15T11:57:00.001-07:00

Technological advancements and the global economy has made innovation and adaptability a critical component for sustainability. Thus we need to push our conventional management model into antiquity and move towards the more dynamic model of self-managing teams.

As we all know, Agile promotes the concept of self-managing teams. This concept has been gaining steam recently but has actually been around for quite some time. W.L. Gore, for example, has been organized around self-managing teams since 1958.

Let’s start with what we have today and how we got here.

Top-down leadership is the prevalent model in most organizations and is ingrained in most of our thinking and behaviors. The current autocratic nature of management was introduced during the industrial revolution. Its ability to churn out vast quantities of manufactured goods is undeniable. Its primary limitation in today’s business climate is that it requires that a select few contain the wisdom, vision, and authority - and the masses execute the plan. This is great for repetitive tasks but less so for innovation and adaptability.

Just as it is irrational to think that we can plan the development of software up front and rigidly adhere to and execute the plan – as if change isn’t going to happen - so too is it delusional to think that adaptability and innovation is borne out of autocracies and bureaucracies.

Innovations are not planned or commanded into existence. Instead, innovations are developed serendipitously. We search for one thing and unexpectedly find another. We need to structure our organizations in a way that promotes the manifestation of the unforeseen.

As with all things, we must make compromises when changing from one management model to another. As Gary Hamel so aptly puts it in his book The Future of Management:

“You can build a company that is virtually error and mistake free. You can build a company that is highly adaptable. But you can’t do both. In this sense, perfection is the enemy of progress”

If I sound like an anarchist who thinks there’s no room for authority, I assure you I am far from it. Leadership is always needed and always sought out. I believe in authority too. I just think that it is ineffectively distributed in the conventional management model.

The requirements for managers in the contemporary world in which we compete are:

Leading by pushing authority, accountability, and reward to small front line teams.
Leading by creating an environment that allows for experimentation, collaboration, and not only self-managing but self-organizing teams.

These steps will encourage creativity, passion, and a sense of community and mission – all of which are ingredients for a dynamic, adaptable and innovative workplace. A workplace built to compete in the 21st century.

Distributed Scrum - Integrating Offshore Resources

2011-10-05T18:56:00.000-07:00

Tranistioning to distributed scrum with offshore resources is hard. It requires significant change in the way teams are run and how members interact. There is also potential for dramatic improvement with productivity and ROI in general.

There are plenty of articles written about offshoring projects, significantly less written on distributed teams, and a trivial amount written on distributed scrum with oursourced and offshore teams.

In my view, the best option when working with offshore resources is distributed scrum. I have faith in Agile as a framework for development. With all of the risks associated with software engineering and the added complexity of team members distributed across the planet, I want control over how projects are executed and the emphasis on continuous improvement.

With that said, here are a few things to consider when considering distributed scrum with offshore resources:

Team and Agreement Structure

Limit resource distribution: The more distributed the team the more complex the work becomes. Limit the distribution of resources as much as possible.
Aim for long-term partnership: When you find the right partner, don't be afraid of arranging a long-term partnership, e.g. 1-2+ years. This allows for you and your outsourcer to concentrate on team building and continuous improvement without distraction of the next agreement. This does not preclude a 30-90 day exit strategy built into the contract if the partnership goes south.
Measurable results: I'm on the fence with this one. It is common to insert milestone-based payments based on deliverables. This is common for any project that is completely outsourced regardless of whether it is software, buildings, etc. But in this structure, any performance incentive is more akin to a performance-based bonus for employees. I keep harkening back to a book I read by B.F. Skinner where he postulated that most incentives are actually disincentives. Agile principles, when adopted fully, make it clear when resources, or outsourcers for that matter, are failing. Payment schemes typically provide incentive to maximize financial reward not necessarily maximizing productivity and effectiveness.

Build a rapport with your outsourcing partner

Stay close at the executive level: The best situation is one where you have access to the executive level of the outsourcer. When you really need something done, you want to be able to reach out at the highest levels.
Find a true partner: It's important that your offshore partner be your business partner too. The organization that you choose should be an advocate for your vision.
Trade players for a few weeks: It cannot be overstated how important it is that the team members get to know one another on a personal level. The affect on a team when dispartely located people finally meet each other face to face and work together for a period of time is stunning.
Maximize overlapping working hours: Both sides should help to extend the overlapping work hours in a day. It's easy to say "your my vendor, you should bend to my wishes". Altough yoru oursourcer may acquiesce, you should expect more to get accomplished when both sides are engaged and working together to solve this particular proximity problem.

Expect Proficient Communication

One language: Pick a single language for communication and make sure everyone one on the team speaks it well. Seems obvious but I've heard of translators being hired to bridge language barriers. With that said, don't be afraid to learn a few words and phrases in their native language. It will help to add character and color to the team.
Good communication skills - Communication is more than language, it's also proactively alerting the team of potential problems, sharing knowledge, willingness to disagree, recommending improvements, etc. As with onshore resources, have high standards with each person's ability to communicate effectively.
Video conference: For good reasons, we don't conduct telephone meetings when everyone is in the office. As humans, we prefer in person conversation. It is important to emulate the behaviour of colocation and video conferencing is the next best thing to holograms. It's a bit awkward at first but becomes very comfortable very quickly.

Treat Offshore Resources Like Employees

Expect profient professionals: Just as with colocated permanent employees, look forward to the right balance of intelligence, creativity, passion, dedication, and all of the other traits we look for when adding to our teams. Don't lower your standards!
Fit people to jobs: The best team is made up of people who are in positions that accentuate their strengths and marginalize their weaknesses.
Career pathing and mentoring: Check-in with your offshore team mates to make sure they are experiencing job satisfaction. Provide mentoring to advance their careers and expertise. Just as with colocated permanent employees, this helps to build a stronger team.
Work/life balance: Expect that your offshore resources have families and lives outside of work.

Be Patient

Coalescing a team takes time: It takes longer for distributed teams, especially when they are offshore vendors, to coalesce. Many people on the team were hired not by you but by your vendor. Don't underestimate this obstacle. Team building is hard enough when everyone works for the same company and is in the same building. This structure is orders of magnitude more complex.
Encourage coalescence: Agile is inherently one of the best methods of coalescing a team. Let that process of direct communication happen naturally. Assuming good quality team members, the only obstacle is you if you act as a proxy between team members.
Facilitation is key: There will be a lot of change immediately and stability can feel very far off. The person who is in charge of building and motivating the team must have good facilitation skills.

Yes, there is risk and some parts of this are down right scary. But don't lose site of the fact that distributed srum work. When it does, most organizations will find themselves fundamentally stronger than they were before.

The Wild West of Mobile Governance

2011-03-22T18:22:00.001-07:00

With many years of network infrastructure experience and evolution behind us, the networks and workstations within most organizations are pretty well standardized. It’s a different story for mobile devices and especially tablets, where the landscape could be aptly described as a lawless wild west.

The Aberdeen Group just published a study called Enterprise Mobility Management 2011: Mobility Becomes Core IT that show dramatic gaps in IT governance with smart devices in general and tablets specifically have been approached from an “adopt now - manage later” philosophy.

Here are some industry averages of companies that have implemented basic security features which in my opinion depicts a frenzied adoption of mobile technologies in lieu of an IT infrastructure to support them:

Lock and Wipe Capabilities: Smartphones 56%, Tablets 22%
Two Factor User Authentication: Smartphone 29%, Tablets 22%
Data Encryption of Removable Media: Smartphones 26%, Tablets 12%

These data are unbelievable. In the Aberdeen paper, the best in class data are at points significantly better than industry averages but certainly still unacceptable. The laggards are abhorrently bad in their adoption of many of the most basic security features.

As software developers, we should view the rapid - and apparently reckless - adoption of tablets as a precursor to an imminent explosion of mobile development internally within each of our organizations.

We are entering an era unlike any that has ever been seen. One that will make the Internet boom look tame and uneventful. Interestingly, all the development we’ve done in the past has prepared us well for this moment in time. The mobile platform is not nearly as challenging as when the Internet was introduced to us.

The difference maker this time around is having a creative workforce with a vision of how a mobile workforce is different than the status quo and what tools are required to leverage their nearly ubiquitous connectivity.

How Does Your App Dev/Support IT Budget Compare to Industry Averages?

2011-03-20T12:02:00.001-07:00

The Gartner Group has done a great job with compiling data related to industry averages for application development and support. Those industry averages are also sliced and diced by vertical.

Industry averages do not indicate best practices or imply the sweet spot for spending for application development resources; they are merely averages for how your peers are balancing resources.

Anyway, here are some interesting stats compiled by the Gartner Group in 2010. There are lots more to go along with these and you can get them here. Do you know how your organization stacks up against your peers and the IT teams in general?

IT Spend Distribution
- 34% of IT spend goes to application development and support resources. 83% of that spend is towards personnel, of which 60% are programmers.
- Of the IT budget for application development and support, 52% goes towards application development and 48% goes towards application support.
Personnel and Work Distribution
- 37% of IT staff is dedicated to application development and support
- 53% of the work distributed to application development support goes to development projects with the balance towards support.
- Work distribution aligned to the SDLC breaks out to 27% requirements/design, 26% development, 32% testing, and the balance being PM and other admin functions.
Development Methodology

Surprisingly, only 14% of organizations consider themselves to be Agile shops. However, there are an additional 31% that use some form of iterative development.
Believe it or not, Waterfall is still king with 55% of organization using that methodology.

Why Wireframing Should be a Required Activity

2011-02-08T15:48:00.001-08:00

The overwhelming majority of users we deal with are non-technical and visual learners. As a result, requirements documents that are heavy on narrative are a poor conduit to understanding the way in which users will interact with a proposed application. By visually manifesting the application prior to development, wireframes put everyone in the best position to win.

Here’s my anecdotal allegory of the problem with requirements documents. When distributing a requirements document, if there are six business users in the room there are probably four that have read the document and two that understand it. Of the two who have understood the document, each of them is imagining a different application in their mind’s eye. It’s not their fault. This stuff isn’t what they do for living and when they become involved it’s on top of their normal day job.

Wireframes replace the written word with visual manifestations of the screens/pages. From a users perspective (an even from our respective) they are significantly easier to understand than dozens of pages of narrative. Software developers and business users live in different worlds and often think very differently. Wireframes help to bridge that gap through pictures that represent a common understanding of functionality. Because wireframes are a visual representation of the application, everyone in the room leaves with the same understanding of what will be developed.

Wireframes should be developed either alongside requirements documents or in lieu of them. They should also be developed deeply and completely to the point where the development team can state at the end of the wireframing sessions:

“This is what we are building – no more and no less. If we are forgetting a feature and it needs to be added later it will likely effect the schedule and budget.”

From my experience, when the wireframing sessions are done well and are complete, change orders are almost never disputed. Don’t get the wrong idea, however, that wireframes result in everyone living happily ever after. Change orders, regardless of whether disputed or not, still frequently cause frustration and require negotiations with scope, schedule, and/or budget.

It’s amazing how often what seems to make sense in thought and speech requires modification once manifested on screen. Wireframes help to alleviate, but not eliminate, this affect. Needless to say, virtually every project succumbs to Humphrey’s Requirements Uncertainty Principle which states:

“For a new software system, the requirements will not be completely known until after the users have used it.”

Wireframes also help to uncover hidden business rules. We all take what we know as professionals for granted and sometimes inadvertently omit information from those with less experience than ourselves. When the development team is discussing the need for each page, workflow, control, validation rule, et al it’s a natural process to hit upon each of the business reasons and processes and discuss them in detail.

Wireframing also helps with estimating projects. I’ve mentioned in previous posts about the inherent problem with estimating software projects. A good example of how badly human beings are at estimating is a statistic from “Demystifying the Black Art of Estimation” by Steve McConnell that most estimates are actually 30% of the actual level of effort. With that kind of epidemic of over-optimism, the clearer we, and the business users, can visualize the application – the better.

Once the wireframes are developed, a finer grained estimate is possible than prior to the wireframes. Each page and feature is able to be enumerated, sized and prioritized. Any prior estimates should be validated against the new information provided by the wireframes.

Wireframing the application doesn’t eliminate documentation. Things like complex business processes still need to be codified.

Wireframes, however, do a better job than requirements documents at articulating for what purposes users need to use a system and the most efficient way in which to use it.

Could the Unthinkable Happen to Microsoft?

2010-12-10T15:30:00.001-08:00

This year has been a pivotal one for the OS business. A few key stats caused my mind to fast-forward to a possibly unthinkable scenario –- Within the next 5 years, Windows will not be the dominant OS, and in fact is set up to be this millennium's OS2.

Yikes.

You’re probably thinking; “How could Jack possibly think that? Has he lost his mind?”

The latter question is hardly up for debate. However, the aforementioned radical scenario is far from certain, but in my view, certainly plausible. Here are some stunning data and some logic immediately thereafter to make my compelling argument. I end the story with all the reasons why Microsoft should maintain their coveted pace in the OS market.

How Microsoft Could Lose Their Place

From a Morgan Stanley study published in April 2010, there are four stats that I think support my dire Windows scenario (not prediction).

1. By 2015, there will be more mobile than desktop Internet users.

2. Social network users have surpassed email users indicating the preference to communicate in the context of online collaboration.

3. People are using their mobile devices more as computers than as phones.

Couple those stats with these by Gartner showing the growth of Android and Apple and the decline of Windows in the context of mobile devices.

Mobile technology is moving at a break neck pace that consumers and businesses alike are adopting virtually immediately. More and more, those mobile users are using their mobile devices in the same way as their desktop computers. With the advent of the iPad and it’s competitors, one can fast forward to a day where a user’s mobile device IS their desktop by virtue of plugging their device into a docking station to enable additional sets of ports, a bigger monitor and/or multiple monitors, a keyboard and mouse, and to charge their battery.

One can also make the argument that over the next 2-3 years, the mobile OS business will trim down through attrition to only a few big players; right now the trajectory favors Apple and certain flavors of Android. A side note … doesn’t Android’s story remind you of Linux where at one point there were lots of players that ultimately trimmed down to a select few best of breed?

With only 4%of the mobile market, is Microsoft too far behind to catch up in the requisite time? Could metaphors introduced by Android and/or Apple provide well known and soon-to-be intuitive features where Microsoft could be frozen out of the mobile OS market and thus by extension the desktop market?

Why Microsoft Is Likely To Maintain Their Place

Here are some reasons why the unthinkable Microsoft scenario may not happen, many of which should be attributed to a colleague of mine who counter-pointed me at every turn as I laid down the potential demise of Microsoft.

Android, by nature of it being open source, is not a single, consistently deployed OS but in actuality many flavors of the same OS with a different feel to each one. By contrast, Microsoft, as usual, will continue to provide a singular vision of it’s Windows mobile OS providing a consistency across devices.

Businesses and consumers alike use their devices to be continuously connected and to easily collaborate with one another. Outlook, Microsoft’s email client, is well known and well liked by it’s users making adoption by primarily email users highly likely.

So you say that with a current 4% market share, Microsoft is too far behind and too late the to the game?

Well, the throw away, appliance-like, aspect of mobile devices means that users will be upgrading to new devices every two years or so. As a result, the ability to enter the market with a better mouse trap and convert audiences from one OS to another is constantly available.

My colleague also makes the point that the rapid nature of new, unforeseen, and game changing devices being introduced every 5 years or sooner requires a constant introduction of new OS’s for the foreseeable future.

We’ve gotten used to Microsoft letting others break ground then usurping control of said ground. However, with the speed at which mobile devices are being adopted and replacing desktops, one has to wonder if waiting has put their place in the OS market in jeopardy.

Dreamforce 2010

2010-12-10T11:43:00.001-08:00

This was my first Dreamforce and I have to say that Salesforce really knows how to put on a show. Nonetheless, it was a good take and worth the money and time. There was lots to see, plenty of people to talk to and a good number of significant Salesforce announcements . So here is my summary as it relates to app dev.

The biggest news is their mega jump into cloud-based application development with the announcements of the acquisition of Heroku, partnership with VMWare to offer the VMForce platform, and their abstraction of the Salesforce database platform into a product called Database.com.

Clearly, Salesforce’s slew of recent offerings is their foray into the competition for cloud-based developers joining Amazon EC2, Google App Engine, and Microsoft’s Azure.

Salesforce is playing catch up. Amazon and Google leveraged their infrastructures for development and hosting services two years ago. Microsoft was talking about Azure soon thereafter and unveiled it for general release earlier this year.

However, with the acquisition of Heroku, a hosted application development platform for Ruby on Rails developers, and their 107K hosted applications, Salesforce has bought a development community. If you are going to be late to the game, they did a great job of catching up.

VMForce, which is a Salesforce/VMWare partnership, was unveiled this past summer and is geared towards the enterprise Java development community. Like Heroku, VMForce is a hosted application development environment.

Both Heroku and VMForce provide developers with the ability to code locally and publish to the cloud.

Database.com is an abstracted version of the Salesforce database platform. The IDE is completely web-based and all development happens in the cloud. Like all demos, it looks pretty slick but time will tell whether it is scalable and its performance is up to the standards that application developers and their customers expect.

Salesforce has bet big on cloud-based application development. It’ll be interesting to see how it all shakes out between them, Google, Amazon and Microsoft.

P.S. A funny thing happened to me on my first day back from Dreamforce. I logged into LinkedIn today and saw a note in my message box from Microsoft offering me a trial version of Microsoft CRM Online. Weird coincidence ;)

Humphrey's Requirements Uncertainty Principle

2010-11-15T14:42:00.001-08:00

Watts S. Humphrey (July 4, 1927 - October 28, 2010) passed away over the last few weeks and as a memorial to him I thought I'd post an article about his Requirements Uncertainty Principle, which is a cornerstone of Agile's approach to defining system requirements.

Watts Humphrey contributed significant thought leadership in the software engineering process and one of the principles he states is requirements are inherently uncertain. To quote and excerpt from his book "A Discipline for Software Engineering":

"This creative design process is complicated by the generally poor status of most requirements descriptions. This is not because the users or the system's designers are incompetent but because of what I call the requirements uncertainty principle:

For a new software system, the requirements will not be completely known until after the users have used it.

The true role of design is thus to create a workable solution to an ill-defined problem. While there is no procedural way to address this task, it is important to establish a rigorous and explicit design process that can identify and help to resolve the requirements uncertainties as early in the developmental process as possible."

Back in 1995 when this book was written (2 years after the founding of Scrum), Humphrey recognized that the software engineering process was broken and that repeated attempts at having a requirements document comprehensively describe a proposed system was met with failure many more times than with success. We are 15 years removed from this publication and there are many development teams still searching for this fictitious Holy Grail!

As we all know, Agile addresses Humphrey's Requirements Uncertainty Principle by:

Capturing what users' want in user stories
At the time of development, collaborating orally and through whatever documentation is required to fully understand what is to be developed
Designing and developing features to address the requirement(s)
Then immediately thereafter providing what's been developed to the user(s) so the requirements can thus be fully known.
Repeat

Why wait till the end of a project or many months after a feature has been developed to get feedback from our users. It is always most efficient to make the inevitable, yet unforeseen, changes to features immediately after they have been introduced.

The Uncertainty Principle in Software Engineering

2010-10-31T17:40:00.000-07:00

Way back in 1996, three computer scientists, Hadar Ziv, Debra J Richardson, and Rene Klosch, wrote a paper that should be better known than it is. It's called The Uncertainty Principle in Software Engineering. Many have shortened the reference to Ziv's Uncertainty Principle. It states that "uncertainty is inherent and inevitable in software development processes and products". This principle sheds light on why Waterfall is well intentioned but flawed as a development methodology and Agile is better suited to deal with the uncertainty in software development.

Ziv's Uncertainty Principle models uncertainty in software engineering using Bayesian Belief Networks. In the software development world, Bayesian nets are most commonly known in its use in search algorithms used on large volumes of text and hypertext.

The authors focused on five areas of software engineering to demonstrate uncertainty;

requirements analysis
transition from requirements to design and coding
software re-engineering
software reuse
software testing

The authors also provided three example sources of uncertainty of which below are my paraphrased descriptions of each:

Uncertainty in the problem domain: The problem, for which an application is developed, exists in the real world. We all know the real world has many uncertainties, many of which are not, and/or cannot, be addressed by the application being developed.
Uncertainty in the solution domain: Building the application itself introduces uncertainty beyond the uncertainties in the problem domain. The example used in the paper is the act of debugging of race conditions from concurrent use. There is uncertainty in the exact conditions that cause it as well as how to observe the condition for reproduction. The authors use Heisenberg's uncertainty principle as similar a affect where by the mere attempt at observing an environment will change it. If you've had to debug a problem with concurrent use, then I'm sure you see this connection.
Human participation: Human involvement introduces uncertainty through business logic built into the application. Business logic coded into an application do no typically address explicit uncertainties. We code mostly based on certainty - not uncertainty.

The point is that attempting to address in advance all of the potential situations and conditions that will be faced in production is futile. The most effective way of dealing with the inevitable uncertainties is to get the application in the hands of the users as soon as possible and let the real world tell us what needs to change. This doesn't mean production alone. It means putting the application in the hands of users who can use the application in real world circumstances.

Waterfall makes the noble, yet inherently flawed, attempt at making the application as rock solid as possible on paper before commencing development. Agile's principle of frequent inspection requires that the team deliver working code frequently. Agile accepts and embraces the fact that once in production, the real world will show that significant imagined truths will be deemed either false or incomplete thus requiring non-trivial modifications and enhancements. These changes are most effectively implemented when features are introduced and not when the application is fully developed.

Another Explanation of Story Points

2010-10-30T07:14:00.000-07:00

Story points seem to be the most misunderstood concept within Agile so I'm going to take my stab at helping others to understand what they are and their importance.

The typical question is: Why estimate user stories with these ambiguous and arbitrary things call story points when I can use hours which inherently make more sense to me?

Hours may make sense over story points at this very moment, but hopefully I'll be able to change your mind after you complete reading this article.

The major impediment to using hours is the variability between people and teams. I recently heard Jeff Sutherland state that a Yale University study has shown that what will take the best developer one hour to complete, the worst developer needs 10 hours to complete. When comparing best and worst teams that grows by an order of magnitude so an hour of the best teams translates to 2000 hour for the worst teams. The variability of hours is a real impediment and is typically a huge time consuming task on most projects.

Story points simplifies the estimation process by taking the developer/team variability out of the process by assigning a level of complexity to user stories. Many in the Agile community use a subset of the Fibonacci sequence as the units of measure; 1/2, 1, 2, 3, 5, 8, 13, 21, 34 and 45 per user story respectively. Think of story points as a more flexible version of assigning estimates as small, medium or large.

The story point estimation process begins by the team selecting the smallest user story and mutually agreeing on its complexity by assigning it a number from the Fibonacci subset of numbers. This user story is called the "keystone", meaning all other user story estimates are based on relative complexity to the keystone user story.

The byproduct of this is that upon sprint completion the team is able to report to the Product Owner in a more concise measurement of productivity - number of story points completed. The Product Owner sees things through the lens of project stories, features, etc. It makes more sense to communicate velocity, i.e. sprint productivity, in the form of story points than hours. Story points translate directly to what is manifested on screen when the working code is demonstrated and hours do not.

Here is an example of using hours as a measurement of sprint effectiveness. A team of four developers work on a project for a two week sprint. At 40hrs per week, the four developers have 320 hours of available time to work during the sprint. However, once the sprint is complete, their velocity shows that they only accomplished 60 hours of work based on the estimated hours assigned to the user stories they completed. That is a misleading depiction of productivity that leads to irrational conversation about whether the team is working hard enough.

Using story points, the team's velocity is expressed in terms of productivity related to the complexity of the tasks completed. For example, let's say the Product Owner knows in advance of the sprint that the team's goal is to complete 35 story points. At sprint completion, the effectiveness of the sprint is measured against the sprint goal and previous sprint velocities. Story point velocity is used to view the overall project productivity in terms of acceleration/deceleration, i.e. is velocity increasing or decreasing.

The Product Owner can also easily translate velocity into an estimated release date more easily than with estimated hours. As an example, if there are 350 total story points remaining in the product backlog, and the velocity of the team is at 35 story points per sprint, then the product backlog will be exhausted in 10 sprints.

Lastly, estimating projects is not only faster but more accurate than estimating in hours! Jeff Sutherland references two experiences related to this in one of his blog articles (read it here).

I hope this clears up what a story point is, why we use them, and provides a compelling reason to transition from hours to points.

Performing a Review of a Software Development Team

2010-07-30T07:31:00.000-07:00

I've had several conversations this week about how difficult it is to provide reliable and consistent software development. Whether the team is an internal IT team, a product team, or a team of consultants, software development is rarely a pretty sight. It is sort of inline with the "making sausage" analogy. However, it doesn't have to be ugly so to help I thought I'd provide a checklist of attributes that every development team should consider.

Development Infrastructure

Each developer should be working in a virtual environment. Whether those VMs should live on the users computer or on the network is a debatable point. I believe the environments should live on the network to ensure they reside in the safest place possible. Residing VMs on the network also removes the dependency of a workstation being online.

Each developers’ local version of source code and folder structure should be identical so if someone were to go from developer to developer the location and folder structure would be identical.

There should be an integration environment to which a build is deployed at least once a day automatically. If the build fails the entire teams should be notified. At least one person should get an email when it succeeds. I’ve been in situations when everyone thought the build was succeeding reliably when in actuality the build machine was down. The integration environment should be identical to the production environment, e.g. web server, app server, database server. When the application is deployed it should tear down the servers to a base environment and rebuild it from scratch. At the very least, this environment provides developers with the ability to test their code in an environment that replicates production. Ideally, there should be automated QA testing that happens with each deployment. It doesn’t have to be full regression testing; an automated smoke test goes a long way.

Continuous integration is a must have. Finding out immediately when a developer has broken the build is vital. It’s so much easier to fix issues as they occur than to untangle them weeks after many developers have introduced their own flavors of bugs.

Development Methodology

This one is a big deal to me because I think very few shops actually have a development methodology. My evidence is anecdotal evidence from being a consultant talking with many clients and prospects and through interviewing software engineers. Most profess to having a methodology which turn out to be more of a set of development procedures than a methodology.

One definition of the term methodology is “the methods or organizing principles underlying a particular art, science, or other area of study.” Organizing principles is the key phrase. Organizing principles are more than a set of procedures. Your methodology should be well understood and those using it should evoke confidence that when the going gets rough that your organizing principles will protect you - as long as you stick to them. Abandoning your organizing principles, in whole or in part, when times get tough results in pain for you, your team, your management and your internal/external customers. In other words – everyone.

Estimation Techniques

My own little survival handbook is called “Demystifying the Black Art of Software Estimation” by Steve McConnell. In it, the author states that studies have shown that estimates are, on average, 30% of what the actual level of effort will be. Thus, know that your developers will significantly and consistently underestimate tasks. If you have never read a book on software estimation then it is a must.

The absence of formal estimation techniques could be the biggest failure point in software development. Communicate estimates in the form or ranges not in single values; e.g. 4-6 weeks. Include all the ancillary tasks such as user documentation, technical documentation, project management, etc. When developing the schedule consider vacations and holidays. And lastly, stand strong in the face of resistance from those who want the job done cheaper and/or sooner. You can negotiate rate, features, etc. but you can’t negotiate how long it’s going to take by magically lowering the number of hours and expecting good things to come from it.

Strong Project Management
A project manager, depending on the organization, is either just another resource keeping track of the project plan, or a leadership role on the project. The latter role requires the project manager to be an oracle and captain at the helm and is the one I prefer. “The captain” reference refers to that person as a significant influencer of decisions and direction. The oracle reference refers to the ability to recognize patterns to identify risks and potential pitfalls. Project management is a combination of art and science. It also requires a strong personality who is willing to give bad news as soon as it is known. This isn’t easy for most people to do and lots shy away from it and futiley hoping things will work out.

Team Composition

We all want the best an brightest. But what’s often overlooked is how the team is composed. Being smart isn’t enough. Your team needs to be filled with smart people who fill specific roles and view the world in a specific way. People can be big picture oriented or not, detail oriented or not, organized or not, ambitious or not, etc. It is vital to architect your teams composition, meaning, defining how many of what types of people you want/need on the team. Teams need a variety of types to be successful. Know what those types are and fill them accordingly.

QA Process

The business should write the test plans and ideally they should be written before development begins. If you are an Agile shop, each user story should have test scenarios. There should be automated QA testing. If there are budget constraints there are free tools out there to leverage (Selenium and NUnit combined with Cruise Control will work fine in most cases).

Stakeholder Roles & Cross Functional Relationships

The business users should be heavily involved. My mantra to customers when negotiating for their time is “the quality of the end product correlates directly with the level of your direct involvement.” Given that they are often making a significant capital expenditure, they tend to get involved to whatever degree they are needed. The reality of the outcome as a result of their involvement, or lack thereof, just needs to be communicated.

Resource roles should be clearly defined at the beginning of the project. There should be no ambiguity with what’s needed from the business users.

Cross functional relationships should be nurtured during and outside of projects. it is important that the business and technical teams know that they are partners and together they will be succeed or fail.

Communication

The glue that hold all of these together individually and as a whole is communication. Everyone should be communicating to everyone often and transparently. When issues occur, make sure everyone knows. I’m not saying that there shouldn’t be a communication protocol. There should be. However, when Engineering disagrees that a bug found by QA is actually a bug, the engineer who developed the feature should have direct access to the QA engineer who logged the bug and the product manager who articulated the feature.

Customers should be involved on a daily basis (with Agile via the daily standup). This makes the weekly status report nothing more than a summary of what the customer already knows. Demo working code frequently to get feedback from the customer. At least one engineer should be in the demo to hear directly from the customer and be able to ask questions accordingly.

Transparency can be scary because we are afraid of the repercussions of allowing the customer to see our problems as they occur. What I’ve found is that customers are reasonable business people who understand the ups and downs of projects. The surprises are what get them angry. Transparency eliminates surprises and keeps everyone in the loop with everything that is happening with the project. Under this umbrella of openness the cross functional team inherently act as a unified team navigating the project towards a mutual success.

Why Cloud Computing Won’t Lead to Dumb Terminals

2010-03-28T08:31:00.001-07:00

It’s easy to fast forward to a time where cloud computing will once again lead to an era of dumb terminals. However, I don’t believe that to be true.

Back when dumb terminals were the norm, the world of computers was quite different than it is now. Most importantly the cost of computer systems was staggeringly higher than it is now and the pace of advancement was much slower.

Moore’s Law is at a point where our individual workstations have processing power equivalent to high end servers of only a few years ago. It only makes sense that horsepower at the workstation provides for a promising future for distributed computing and at some point the ubiquity of grid computing. Leveraging millions, and even billions, of computers as opposed to a much smaller set of monolithic servers is a more likely model.

A server-based model provides for central points of of failure; distributed and grid computing does not.

With cloud computing, the proximity of our data is less important (and likely to be less obvious) but it does not necessarily point to a future where my laptop plugs into the network as a throw back to mainframe days.

I believe it actually points to a time that compares more closely to peer to peer computing, similar to Groove’s replication of data across a network of computers. This eliminates central points of failure, promotes greater power (horsepower and volume) by leveraging large numbers, and is more conducive to offline computing and synchronization.

We are all guessing at this early stage of cloud computing, on which every server and workstation is merely a node, so I’m interested in your perspective.

The Proximity Problem: A Cloud Fable Part II

2010-03-10T09:42:00.001-08:00

The Internet has provided opportunities to advance efficiencies like no other time in human history. With proximity becoming less important every nanosecond, collaboration between parties half way around the world is now possible on an on-demand and real-time basis. I’d like to think altruistically and see the world as a utopian Garden of Eden where everyone is working together for the betterment of mankind, but alas, that is not reality - especially in the trade of underground contraband.

Manuel is part of a South American drug cartel. He and his peers view their operation like any other multi-national conglomerate. The objective is to increase revenue and profits. Just-In-Time Manufacturing is as real here as in any other enterprise. The concept of Kanban, the infamous innovation from the Toyota Production Process (TPS) that has been replicated in so many industries is a fully blown implementation in Manuel’s operation.

The United States, Manuel’s monolithic neighbor to the north, is not only his biggest and most profitable region but also his most tenacious and sophisticated foe. Travel to the US for business meetings has never been risk free. Since 9/11, however, the risks associated with in-person business meetings whether in the US or at home, have increased dramatically.

Like any high performing enterprise, Manuel’s is innovative and constantly striving to stay ahead of their competition. With the risk of international travel increasing for both Manuel and his customers, the cloud has been a boon to removing proximity as an attribute of doing business.

One innovation is the use of video conferencing as a way of entering into business agreements between Manuel and his customers. Not only does video conferencing mitigate the risk of in-person meetings but also provides the byproduct of recording the meetings thus acting as a binding contract between the parties.

When choosing his provider, Manuel knew that he could not choose one from the US. The Patriot Act took that option off the table immediately. So he opted for an off-shore provider. One domiciled in a sympathetic nation. In addition to the typical criteria such as functionality, bandwidth, and up-time, Manuel also considered things like extradition and privacy laws of the host nation.

Manuel’s organization embraced the cloud in many ways beyond video conferencing. Coupled with video conferencing, his cloud-based providers for storage, ERP, CRM, email, and banking, Manuel could conduct business anywhere and with anyone. Moving his operation was limited to moving people – not infrastructure and data.

What Manuel did not, and likely could not, consider was the complete infrastructure of his providers.

Cloud-providers potential Achilles heel is redundancy and failover. Customers of cloud providers expect near 100% up-time. The best providers fall into the Five-9s category (99.999%). To provide that kind of uptime there needs to be a sophisticated infrastructure with the same goal as Manuel – minimize proximity as a central point of failure. What Manuel did not know was that several of his providers contracted with US firms for redundancy, load balancing, and disaster recovery.

The one flaw in Manuel’s planning has now come home to roost.

The US DEA has indicted Manuel and a Chicago-based group, which was made up of US citizens, green-card holders, and illegal aliens, for illegal drug trafficking. After years of undercover operations, wire tapping, and packet sniffing, the DEA uncovered a recording of a video conference contract from a data center in Tucson, AZ. The Tucson data center was a tertiary load-balancing location for Manuel’s Chinese conferencing provider. Apparently Manuel struck a big deal over the Chinese New Year when many Chinese expatriates from around the world used his provider to video conference with their families back home. The provider experienced a high volume of calls when Manuel entered into these negotiations which kicked in the tertiary site in Tucson based on call volume and proximity of the callers in the system at the time – the Tucson data center was the most central of those data centers between Chicago and Manuel’s Venezuelan villa.

The DEA has its work cut out for it. Manuel is busted but his nation is unfriendly to US interests. The domicile of the video conferencing provider is likely to be equally uncooperative. If the US is lucky enough through interrogations or packet sniffing to intercept data related to Manuel’s other cloud-based providers, it’s likely to encounter the same minimal level of cooperation from those providers and their home nations.

However, Manuel has an equally uncomfortable situation. His epiphany is that he has no idea where all his data are stored and by the very nature of the Internet there is likely no way to remove it from the locations that can work to his disadvantage. His proximity and flexibility advantages are to an unknown extent countered by his lack of control over those environments.

Going Rogue: A Cloud Fable

2010-03-05T08:13:00.001-08:00

The leadership within a certain division, let’s call it the BizDev division, of a fortune 500 company is entrepreneurial by nature and thus results oriented. BizDev is impatient with the red-tape required to get anything done. Capital expenditures are difficult to get approved and initiatives are equally difficult to resource, initiate, and complete. The leadership’s perception of their peers within the divisions from which they consume services is that because of resource constraints they have a basic reflex towards being obstructionists.

Being entrepreneurial, innovation comes naturally to the BizDev leadership team. They contemplated bypassing the corporate infrastructure and protocol for the greater good of their group. The BizDev team looked to the cloud for solutions to their bureaucratic woes. Using operational instead of capital expenditures for outsourced cloud services made sense to them. Implementing cloud solutions is quick, the process is simple, and if they stay mainstream, the vendors will be known quantities with effective and reliable track records. Why trudge the internal landscape when you can subscribe to existing solutions that satisfy the 80-20 rule. BizDev decided to roll the dice; guessing that the monolith didn’t have the appetite to stop them

BizDev was going rogue.

BizDev specializes in content management and as such has significant storage requirements. They looked to Infrastructure as a Service (IaaS) providers to solve their problem. Storing content at an IaaS provides them with the ability to increase and decrease their requirements in a self-service kind of way with virtually no red-tape. The BizDev team saw this as a no brainer and after a relatively quick selection process contracted with a vendor and began using their services shortly thereafter. The IT division with the company got wind of the cloud deal and although outwardly condemns the move doesn’t fight it and is actually relieved to get BizDev off their back.

BizDev’s gambit paid off. Over the course of several years, BizDev’s use of content management providers became more sophisticated by using specific vendors based on cost and functionality related to specific media. They also expanded its use of the cloud by encouraging its thought leaders to blog using the blog space of their choice and by actively using as many of the available social networking sites as possible.

Word of BizDev’s success in the cloud spread throughout the organization to other divisions. Many using the cloud for their own purposes; marketing, software development, legal services, ERP, CRM; you name the cloud service offering and it was being used somewhere within the company.

Although there was apprehension about the lack of governance over the use of cloud providers and the policies around their use, there was no appetite to take on a task that most anticipated to be a politically charged and multi-year project involving many resources. The use of cloud services was widespread and for the most was providing huge benefits with regards to speed to market, flexibility, and cost reduction. The cloud was considered a boon to efficiency that dramatically improved the organization.

Until the indictment.

The SEC came down hard on the company for alleged indiscretions related to business deals, profit forecasts, and accounting schemes. In addition to emails related to the executives of the organization, the opposing counsel is asking for all the marketing material related to BizDev and other business units to determine if fraudulent claims have been made about product effectiveness.

The company’s lack of governance over cloud resources is now making the process of providing discoverable resources to opposing counsel within federally mandated time frames untenable.

Where does all the marketing content reside? How do they capture what’s required and filter out what isn’t? Are the cloud providers prepared (and willing) to provide what’s requested by the opposing counsel? What are their retention policies and do they follow federally mandated regulations?

Have the respective business units maintained compliance with corporate policies related to external marketing? Is there a controlled and comprehensive set of messaging that’s contained in blogs and articles written by thought leaders? Are there any incriminating status messages posted by overzealous thought leaders in their favorite social networking sites? How does the company search for, retrieve, and provide social networking status messages for key individuals?

Although the cloud is becoming, and will continue to be, a direction for cheap and scalable solutions to common business problems, without governance business entities potentially put themselves in tenuous positions. The cloud is just another option like the myriad of other options for conducting business operations. And as such, cloud vendors should fall under the governance of corporate policies like any other vendor.

Which Cloud is right for you?

2010-02-20T11:53:00.001-08:00

Deciding which cloud is right for your software engineering team isn’t as difficult as you would think. The big players getting all the attention in the media today are Google, Amazon, Microsoft, and Force.com. For most software engineering teams, however, none of them is the best option from a development perspective.

The big four are geared more to production environments than development. Although most of them provide the ability to extend their environments through local IDEs like Visual Studio and Eclipse, that’s hardly enough for a development team. They don’t address things like integration, testing, and demo environments.

A good example of the inadequacy of the big four as software engineering environments is the fact that Amazon deletes VMs when they are shut down. That’s clearly indicative of a public facing focus where shutting down a VM equates with shutting the doors on a business. Thus, Amazon, as well as Google, Microsoft and Force.com, are not options when moving “development” to the cloud

A more appropriate approach for cloud development environments are Infrastructure as a Service (IaaS) providers geared specifically towards development teams. They are lesser known but I assure you they are out there.

Under this structure, environments can be spooled up on demand. Costs are controlled by having VMs running during working hours and shut off during off hours. In addition to conventional application development, teams need temporary plug and play VM infrastructures to be used for an finite period of time to develop POCs, demo systems, troubleshooting specific issues, and many other one-off situations.

My firm’s application support service offering is an ideal function that is conducive for IaaS providers. Our development team has 40 VMs consisting of various client applications and project infrastructures. Putting all of them in the cloud with the ability to start and shutdown on demand based on client requests provides our team with the ability to focus on our core competency (software development) and offload those functions that aren’t our core competency (infrastructure engineering and management) to cloud providers.

IaaS providers aren’t for everyone either. They are ideal for organizations with lots of infrastructure needs. They are typically not suitable for public facing production environments nor are they ideal of a single demo VM or similarly smaller infrastructure.

In general, here are the rules of thumb:

Big Four: If you have a public facing product with a high hit rate, then one of the big four providers is likely to be the right choice.
Infrastructure as a Service (IaaS):

If you are a software development team with many sophisticated infrastructures to support many projects and would like to offload the infrastructure supporting the SDLC, then an IaaS provide is the right choice.
If you are a startup product company and do not want to invest in infrastructure engineering resources, then an IaaS provider is worth considering.

Internal Resources: If you are a small application development team with few environments, your internal resources or that of your consulting vendor will be the most efficient solution.

Evolution of Software Development and Cloud Computing

2010-02-06T14:11:00.001-08:00

Cloud computing is a relatively new term for what has been around for a decade – leveraging virtual environments within the firewall and beyond it. Many software engineering teams have been in the cloud since its inception and will likely be drivers for its adoption going forward.

HOW DID WE GET HERE?

Prior to virtual environments, the infrastructure to support development efforts was physical. Thus, software engineers had their development tools loaded on their respective workstations and integration, test, and production environments were made up of servers on the network. Coupled with the expense of physical hardware for every required infrastructure environment, installing development tools and custom applications on workstations and servers often had a corrupting effect on those machines.

If you subscribe to the definition of a cloud environment as being any that is virtual – even those behind the firewall – then the first iteration of a private cloud for software development came with the emergence of using virtual environments for development workstations. This had a mammoth effect on development efficiencies and IT governance.

As an example, in 2001, I was hired as a consultant to help a client develop the next generation of their commercial e-Discovery product. The suite of applications were complex with many moving parts and technologies. It took the average developer 1-2 weeks to get their development environment up and running. Because of the expense associated with physical infrastructures there were limited environments to promote code, and continuous integration wasn’t even a dream.

I was hired back again in 2005 for yet another next generation development effort. This time, the developer workstations were based off of a base VM. Getting up in running took hours instead of weeks. The fact that developer tools were loaded on the VM instead of the host machine was a dramatic improvement for IT. At last, a software engineer’s machine was like everyone else's in the organization.

In addition to developer workstations, the integration and test environments were also virtualized. This allowed the release engineers to revert environments to their base snapshots in preparation for new releases. When we needed to branch testing, we would spool up another virtual environment. Our only limitation was the hardware on which those environments were deployed.

WHERE ARE WE GOING?

The last several years have seen virtual environments that exist beyond the firewall. The current ‘big four’ front runners are Google App Engine, Amazon Web Services (AWS), Force.com by SalesForce, and Microsoft’s Azure platform. These providers offer scalability for large volume web-based applications with built-in clustering and load balancing.

However, the big four aren’t inherently development environments with sophisticated processes for building and testing applications and maintaining source code control, build scripts, and other requirements of complex projects. For these types of projects, you would use Infrastructure as a Service (IaaS) providers. IaaS providers allow teams to customize virtual server and workstation configurations in a plug and play fashion that’s conducive to on-going development.

A complete cloud solution would be to combine an IaaS provider with one of the big four. The IaaS provider would be the infrastructure for the development and test environments. Releases would be published from the IaaS provider to one of the big four for your high traffic web app.

Software Engineering teams interested in using cloud providers need to decide which flavor of Internet-based cloud infrastructure is appropriate for them; extending a developers workspace and “being” the developers workspace.

Each of the big four allow developers to use existing IDEs to extend their development environment. Here are the major IDEs and the providers support.

Visual Studio: Azure
Eclipse: Google App Engine, AWS, Force.com, Azure
NetBeans: Google App Engine, AWS

Each of the big four support one or more programming languages. This list below is not comprehensive but provides the major languages:

Azure: C#, Java, PHP, Python
Google App Engine: Java, Python
AWS: C++, C#, Java, Python
Force.com: Active Script, Apex, Visual Force

As you can see above, which platform you choose is dependent on your internal expertise. Force.com is the platform that does not provide support for the major languages. However, it’s a more rapid development environment than the others – somewhat like a 4 GL – and I anticipate that it will be a player for the long haul.

The appropriate infrastructure associated with the decision to move to a cloud provider is a decision to think through carefully. The various configurations are virtually unlimited, e.g. have the entirety of your virtual environments either behind or in front of the firewall, split environments between behind and in front of the firewall, and deciding which, if any, segments of the infrastructure are public.

The adoption of cloud providers is a non-trivial decision as is the design of the appropriate infrastructure. However, as a manager, software engineer, and consultant, the cloud is likely to be a catalyst for a trend that offers developers an opportunity to think more about development and less about infrastructure. Which is as it should be.

Will Google Wave Re-Define the Blog Paradigm?

2009-11-26T05:22:00.001-08:00

As we all know, blogs are a social networking tool to desseminate information and capture feedback. The tools currently in use almost provide the mechanism for discussion but don't quite get there. Google Wave holds promise of bringing blogs to the conversational promiseland - at least in the world of typed messages.

Currently, the process works like this: Someone writes a blog article that is typically several paragraphs long with several points linked together by logical reasoning. After visitors read the article, they input a comment at the bottom. The comments usually consist of general commentary on the article subject matter, specific commentary on one or more points within the article, or extending the article through logical relationships to subject matter not discussed in the article.

Commentary from a blogs reader ship currently consists of narrative pointers into the article, e.g. "I disagree with you when you state...". These narrative pointers become exponentially more cumbersome with high volumes of visitors and comments. Discussion is lost because it quickly becomes impossible to follow the massive array of narrative pointers contained in the set of disparate comments.

Google Wave blows up this paradigm by allowing inline commentary within the article. Comments are no longer aggregated by visitor containing multiple points at multiple locations within the article. Comments become decentralized and are input directly in the article at the location where the specific points are made. This allows follow-on visitors to add to the discussion on just those points with which they have interest and absolved of the prerequisite filtering of content and comments that currently exists. The playback feature is also a pretty cool way of being able to read the article without comments and follow visitor feedback sequentially as they occurred.

I posted much of the text from this blog to a wave that I made public. Virtually immediately after posting it, I started getting inline feedback from a Wave user. If you have a Google Wave account you can view it by searching for with:public google notarangelo. There's some interesting commentary.

If you want to contact me using this new and interesting medium - and what I think is the future of blogging - then get a Google Wave account (I have some invites available if you need one) and ping me jack.notarangelo@googlewave.com.

You can read more about Google wave at http://wave.google.com/.

Economic Darwinism

2009-11-10T06:16:00.000-08:00

Over the summer, I wrote an article in this space called It's Winter in July. The premise being that in these tough economic times everything you've done to this point in your career has culminated to your current market value. This article extends that subject in the form of what I call Economic Darwinism. Surprisingly, my Google searches on the term came up with hits but not in the way that I think of it.

Economic Darwinism in my mind is related to survival of the fittest. We have been in this downturn for over a year and it's been difficult. Lot's of layoffs, attrition, hard looks at the way we do business, and in many cases lots of change.

For the past year or so you have been in survival mode which is a good and healthy process. If you have survived being in survival mode then it likely means that you are built for what you are doing.

When you look around at your respective team members you likely see a strong set of individuals that as a group can deliver whatever needs to be delivered. In a culture of meritocracy, which for the most part describes software engineering, we have been transformed into lean teams with a kick-ass set of players that gets shit done. Yes, it was painful getting to this point, but the result is good.

If that accurately describes your situation, then from a leadership perspective the view should be "The recovery starts with us. The inevitable new phase of growth will be through this team. And through that inevitable growth there are opportunities for everyone."

Morale, whether poor or euphoric, is a state of mind. The thoughts bouncing around your cranium, regardless of whether they are verbalized or not, influence those whom you lead. It's your belief system that will - and does - significantly influence whether your team feels confident or insecure. Look around, see the strength of your team, and know that the recovery starts with you. If you believe that, confidence/morale will increase. If you choose otherwise then you should expect the malaise to continue.

Let there be no doubt that which ever side you land is all in your head. Your harvest is in part dependent on the seeds sown now in the Spring of this recovery.

Cloud Computing: Hype vs. Reality

2009-10-26T04:17:00.001-07:00

The term cloud computing is a recent branding effort for an umbrella set of hosted service offerings. Of course hosted offerings have been around for decades - SalesForce.com has been providing an Internet-based (i.e. cloud computing) CRM solution since 2000. There are also a plethora of other vendors with offerings related to Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and Software as a Service (SaaS).

Cloud configurations fall into three categories – Public, Hybrid, and Private.

Public: Public cloud offerings are those that are typically subscription-based where all of the hardware and software purchases and maintenance are abstracted away from the customer.

Hybrid: Hybrid cloud offerings are provided with a combination of abstracted hardware and software with private infrastructure configurations. It is presumed that this will be the most likely configuration as cloud services become more mature and mainstream.

Private: A private cloud is an Internet-based offering where the entire infrastructure is managed by customer personnel and typically has security applied to limit usage to authorized personnel.

There are some who will refute my definitions above as too limited because they consider on-site virtualized environments as a valid cloud configuration. In my view, if the service isn't Internet-based then it's not in the cloud. Time will be the arbiter of this amorphous aspect of cloud computing.

So why has something that has been around for so long now become worthy of re-branding? The major drivers are the ubiquity of the Internet, the reliability of Internet connections, the maturity of web-based interfacing technologies, and the rapidly expanding blur between desktop and web.

The remaining obstacle for most organizations with moving to the cloud model is security.
Moving sensitive data and intellectual property to a cloud provider is a risk to be considered carefully. An internal infrastructure can be completely isolated from the outside world where malicious activity is voracious to say the least. In fact, it's safe to say that because of intellectual property and the sensitive nature of the data being stored, that for some companies there will always be a need for an internal infrastructure. Government regulations regarding privacy and organizations obligations to ensure protection of personal data will continue to be a driving force for internal resources.

There is also the counterpoint that because of perception and the fact that their services are in the cloud, cloud vendors focus significantly more attention to security than many internal IT teams in organizations where IT is not the core service. That is a valid point that deserves consideration.

You could also make a case that cloud computing implemented as the status quo across all industries and organization size - in its current form – could be considered a national economic risk. As cloud computing proliferates and third parties develop widgets of functionality as a subscription-based model, resources from disparate cloud environments will become interwoven into applications resulting in nested dependencies. When developing the next killer application, why invent a wheel when you can subscribe to one and integrate it into its processes? A benign or malicious act that triggers a negative event across the Internet could have significant economic consequences depending on its impact. There could also be inherent compatibility issues when cobbling unrelated cloud-based widgets together.

Cloud computing has also been touted as the return to dumb terminals with all software being cloud and subscription-based. I can not see that happening in the short to mid-term. Much of this article was written on my vacation without wireless Internet access. My writing was done offline using software local to my computer. Installing binaries locally is going to be a way of life for the foreseeable future albeit possibly a bit neutered in some instances since more and more of what appears to be local functionality in many applications is actually sourced from the web. (e.g. online help, lists of templates, installing add-ins)

If you are reading this because you are interested in cloud computing and are considering how to leverage it for your organization then you are ahead of the curve. Take your time, develop a long-term strategy (including governance policies), implement in controlled phases, and seek advice from professionals to ensure your implementation is the right one for your organization.

Social Networking, Privacy, and Identity Verification

2009-09-28T16:25:00.001-07:00

The value of total obscurity on the Internet is overestimated and is more destructive than constructive. Our fears about privacy infringement are largely irrational; at least in the context of social networking.

The ability to be totally obscure allows those bent on malicious intent to cloak themselves in an anonymous or fake identity – allowing them to say anything they want, about anyone they want, and without any repercussions.

It’s still rare, but sites are starting to require identity verification and that trend, in my opinion, is certain to continue. It has to. Identity verification provides for accountability and enforces a set of socially acceptable rules of etiquette just as they exist in the real world.

Most of us aren’t afraid to have our identification verified for credit cards, licenses, loans, job verifications, and many other situations. The large majority of us willingly provide our credit card information to purchase goods over the Internet.

Is caller ID a bad thing? Don’t most of us frown upon those that restrict the rest of us from seeing their identity when they call by setting their caller ID to private?

I just joined a social networking site called BestThinking.com that requires subscribers to have their identification verified. And you know what? I was not only willing but found it refreshing!

They do it in a very clever way with little personal info - nothing that you would consider a security risk to divulge – and a short multiple choice quiz of information about you to make sure you are really you. The bottom line is that social networking sites where those involved are willing to be known by their real identity adds credibility to discussions and opinions.

Remember the sense of foreboding people had when purchasing goods over the Internet was first introduced? As time went on we realized that our fears were irrational. Yes, there is identity theft and sometimes on a colossal level. That is what comes with forging a new frontier. It takes time to get it right. In spite of that, we continue to purchase goods over the Internet because we consider there to be considerably more value than risk.

We are now forging forward further into the Internet as a new frontier where a new level of identity verification will become increasingly more required and hence a more credible place to share information.

Do I think we should encourage legislation to force identity verification as a prerequisite to participating on the Internet? That would be a hearty No.

I do, however, think identity verification will become ubiquitous within this coming decade for the sheer reason that deep down we want to mold the virtual world into one that resembles, as closely as possible, the physical one in which we live.

Why?

In addition to basic familiarity and the comfort that brings, the physical world is more soundly implemented; especially with regards to established rules of engagement. The Internet adopting attributes of the physical world, which have been tested, modified, and enhanced over many millennia, will help to realize the potential of the virtual world. Identity verification brings the Internet one step closer to maturity and in this corner it’s a welcomed step.

I'm interested in your thoughts.

The Future of Crowdsourcing

2009-09-12T07:47:00.001-07:00

Crowdsourcing, for those who are new to the term, is the concept of entities outsourcing problems to a large and typically unrelated group of people. For example, Netflix has an ongoing competition - open to anyone - to develop an algorithm to predict customers ratings based on their past ratings that bests Netflix's proprietary algorithm by 10%. The prize is a million dollars.

Crowdsourcing is not a new concept. The Longitude prize was an open competition established by Great Britain in the 18th century to solve the maritime problem of discerning longitude at sea. John Harrison was awarded the prize for his invention of the Chronometer. He wasn't treated well either. The dude solved the problem and was delayed the prize money for a full 30 years.

John Harrison's poor treatment by Great Britain illustrates one potential problem with crowdsourcing however there are others - little to no contracts, lack of continuity with contributors, potential lack of interest thus little to no participation, low to no wages, and risk of malicious intent.

The global recession has resulted in rapid growth of crowdsourcing due to two major factors:

It is often cheaper for companies to crowdsource solutions as opposed to directly hiring or contracting with professionals.
There are lots of people out of work so the pool of willing participants is high.

When the economy turns around will the resources currently involved in crowdsourcing dry up? Will the competition for intellectual property and time to market pressures move corporations back to more traditional methods that are more easily managed?

My answer to both questions is no.

With corporations having the ability to tap into a world population for ideas and solutions there is bound to be better results than with a small set of specialists. Crowdsourcing offers the possibility of tapping into brilliance without having to interview for that special person who will develop that next killer product.

As far as the contributor is concerned, crowdsourcing offers recognition, flexibility, collaboration, pay, and other self-satisfying attributes. The city of Los Angeles provided a survey to it's population asking questions such as "What services should be cut to balance the budget?" with a list of city services from which the constituent may choose. This type of crowdsourcing relies on non-monetary rewards but still has a high rate of participation.

One of the more interesting things to consider is how crowdsourcing will effect various occupations such as those in the creative design industries. When creativity is outsourced to the world, there is a potential for deleterious effects on wages and the number of permanent positions in those fields.

In my opinion, crowdsourcing, like social media, is in its Wild West phase. There will be significant movement and change along the way and its current incarnation will be unrecognizable 5-10 years from now.

As crowdsourcing models mature and become easier to manage the majority of us will be involved in some sort of crowdsourcing as an inherent part of our lifestyle. Just as I continue to manage my LinkedIn contacts and update my status on Facebook, I will likely also be contributing to my favorite crowdsourcing activities.

Kano Model for Prioritizing User Stories/Requirements

2009-09-01T10:02:00.000-07:00

Regardless of whether you subscribe to Waterfall or Agile as your preferred methodology, some form or prioritization of requirements (Waterfall) or user stories (Agile) will take place.

One of the more difficult tasks is helping the user community to determine the varying degrees of importance of each requirement. One method to help with this process is the Kano Model.

The Kano Model is named after Professor Noriako Kano who in 1987 developed a theory for product development to classify features into five categories based on answers to questions about the specific features. Following are the five classifications.

Attractive	Delighted to have but unexpected.
One-Dimensional	Features customers compare with your competition.
Must-Be	A must have feature.
Indifferent	Neutral about the feature.
Reverse	The customer do not want the feature and actually expect the reverse of it.
Questionable	Indicates that the customer is unclear about the nature of the feature

The features are classified by asking the customer two questions – one functional and the other dysfunctional – to which the customer selects one of 5 possible answers.

The questions are:

How would you feel if the feature was present in the product?
How would you feel if the feature was absent from the product?

The answers from which they may choose are:

I would like that.
I require that.
I don’t care about that.
I can live with that.
I dislike that.

The initial reaction of most people when posed with the functional/dysfunctional question think “won’t the questions offset each other?” As it turns out most often they don’t.

An excellent example that I’ve heard in the past is with a milk carton that has a thermometer on the outside so the customer can see what temperature the milk is. One may select answer 1 I would like that as the functional answer but select answer 4 I can live with that for the dysfunctional question.

The answers are then compared to a matrix below to arrive at the classification. The letters in the middle of the matrix represent each of the classifications via its first letter, e.g. A=Attractive. To take our earlier example of the thermometer on the milk carton, the functional was Like and the Dysfunctional was Live With thus the matrix indicates that the classification is A=Attractive.

	Dysfunctional
		Like	Expect	Neutral	Live With	Dislike
Functional	Like	Q	A	A	A	O
	Expect	R	I	I	I	M
	Neutral	R	I	I	I	M
	Live With	R	I	I	I	M
	Dislike	R	R	R	R	Q

With the Kano Model, one is able to ask the customer two short and concise questions and ultimately gather critical data regarding the importance of the feature to the product. As a result, a development team can easily discern what’s mandatory, what’s nice to have, and where the land-mines are.

There are additional techniques that can be used in combination with the Kano Model to further prioritize requirements like weighting features but alas that will have to weight for another day.

Estimating Defect Production

2009-08-25T04:40:00.001-07:00

Estimating the number of defects a project should expect to produce and remove is one of the least talked about subjects in software development. In this article I'll provide some industry statistics to help with that estimation process and hopefully convince you of why Agile will likely reduce the level of effort associated with defect fixing.

There have been studies performed to estimate the number of defects that a software development effort will likely encounter during its life cycle. Obviously the larger the project the greater number of defects one should expect to encounter. If you follow this blog, you’ll know Steve McConnell’s book "Software Estimation: De-mystifying the Black Art” is one of my favorites and is always by my side. One of the chapters discusses estimating defects. In his book, McConnell references a Capers Jones (2000) study that indicates a reasonable expectation is to have 50 defects per 1000 lines of code (LOC).

However, a more granular look will show that smaller projects will experience fewer defects than larger ones; for example a project with fewer than 2K LOC will likely have 0-25 defects per 1K LOC where a project with over 512K LOC will likely have 4-100 defects per 1K LOC. Keep in mind that factors such as your programming language and other technologies will effect this estimate. It’s always more accurate to use historical data to estimate effort but in lieu of that, these data are better than nothing at all.

Here are a few more factors to consider:

- Defects occur at all points during development e.g. requirements, architecture, development, documentation, etc.

- There are best practices for defect removal such as design reviews, code reviews, prototyping, unit testing, system testing, and various levels of beta-testing – each of which have different removal rates. The highest removal rates come from formal code reviews (45-70%), prototyping (35-80%), and high volume beta testing (60-85%). Surprisingly (at least to me) one of the lowest removal ratings comes from regression testing (15-30%).

Here is where I transition from conveying data to providing my opinion on why Agile allows a development team to reduce the effort associated with defect removal when compared to Waterfall.

With Waterfall, the life cycle stages occur in a sequential manner,i.e. requirements, design, development, and test. Although it could be reasonably argued that the number of defects may not significantly change between Waterfall and Agile I think it is unreasonable to assume that the effort to remove them will remain the same. Defects introduced during the requirements gathering and design stages that are not found until development or even later have a snow ball affect because they pile up on one another.

Because of it’s iterative nature, Agile allows for requirements, design, and development defects to show themselves virtually immediately after they have been introduced. Dealing with defects introduced over a single two week sprint is a lot easier to manage than untangling a slew of defects that occurred over several months. We’ve all been on Waterfall projects where our integration testing revealed flaws that have required rewriting methods and even entire components. I contend that those types of wholesale defect removal efforts are mitigated substantially through continuous integration, daily stand-ups, two week sprints with customer tests immediately thereafter, as well as the other feedback loops inherent in Agile.

I’m interested in your thoughts.

The Balance Between Talent and Team

2009-08-04T07:32:00.000-07:00

I came across an old Joel on Software (Joel Spolsky) article from 2005 called Hitting the High Notes that I thought was particularly thought provoking. One of the premises of the article is that a single brilliant developer is more valuable for innovation and invention than an army of mediocre ones. His reasoning is that the brilliant developer is capable of thinking things and creating things that are virtually impossible for mediocre developers. The best line that captures the essence of the article is "Five Antonio Salierie's won't produce Mozart's Requiem. Ever. Not if they work 100 years."

But how often does the average organization require the elegance and brilliance of Mozart to compose the software equivalent of his Requiem? When viewed from the context of what the overwhelming majority of software engineers do every day, the answer is “almost never”.

Like Joel Spolsky, I’m a big fan of greatness. Real life accomplishments can often be better than fiction. However, software is as prolific and ubiquitous as there are people and it is because it doesn’t require Mozarts to create useful software to serve an organization and save tons of time and money and provide insight into business and industry trends.In fact, the effect of hiring purely on programming talent is at best problematic.

Let’s agree that software engineering is intellectually demanding. You can't be a dolt and do this job effectively. Technologies and subject matter change too quickly so having intellectual horsepower is a must. It’s also my opinion that for those same reasons one needs to understand that being a software engineer is not a 9-5 job.

As an organization, having intelligent and committed individuals is still not enough. The New York Yankees proved that by trying to buy World Series championships year after year through the assembly of the most talented players alone and failed miserably (As a Red Sox fan I consider that a success story).

That talent-alone strategy doesn’t work in software engineering either. The whole doesn’t necessarily have to be more than the sum its parts to be effective but in those cases one should expect mediocrity and not a whole lotta fun for those involved.

To accomplish cool things, if not world changing things, and have fun doing it there has to be synergy. Included with talent, intellect, and commitment, everyone should respect one another as people and professionals, understand what each other needs to be effective, compliment each others talents, and together fill all pieces of the engineering pie. If you think getting all these pieces to fit together properly sounds really, really, hard - I'm in full agreement!

Accomplishing the assembly of the aforementioned team is certainly a non-trivial task. As with everything in life, it requires compromise and balance. In this case, the compromise and balance is between the individual and team attributes needed, weighting them accordingly, and then hoping you’re right when hiring your next engineer.

It's Winter in July

2009-07-27T06:24:00.001-07:00

Aesop's fable of the ant and the grasshopper still holds true – you better prepare for winter or you’re screwed. In case you haven’t noticed we are in an economic winter, my friends, and either you have prepared for it or you have not. It's really that simple.

These are difficult times - no doubt. With the economy the way it is, people will often feel fear and insecurity about their positions. If they've been laid off they may be concerned about when, or even if, they'll find another position.

Admittedly, sometimes being in the wrong place at the wrong time can result in losing one’s job. However, more often than not, the security we have at our current employer, and even the difference between having a job and not, is based on our present value-add to the organization. Your entire career has culminated to this point in time and the perception of your relative importance to the success of a team and organization is essentially set. It was built over the course of years both at this current position and every one before it.

I’ve been a solid ant for a number of years now. Unfortunately, I’ve had to learn hard lessons as a result of embracing my inner grasshopper. The repercussions of being cavalier about an important matter as the care and feeding of one’s career will eventually result in pain. Those grasshopper times, however painful, turned out OK for me and if you are experiencing a harsh winter it will likely be OK for you too. The trick is to constantly reflect on whether you are thinking like a grasshopper or an ant – and know and accept the consequences!

Over the course of a bunch of years I’ve come to realize that no one is entitled to anything. Everything you have and is dear to you is on the table. So have a vision, be pragmatic, take risks, and most certainly, be assertive with your career. If you do those things well you will not only best protect that which is now yours but you will grow as a person too.

Natural Leaders

2009-06-24T04:11:00.001-07:00

Did I intend for this space to be used for discussions about leadership? No. However, the natural risks and pace of change that are inherent with software engineering makes leadership incredibly vital and high quality leadership hard to come by. When I came across an article written by Gary Hamel called How to Tell If You're a Natural Leader I immediately thought that it was important to share it here.

What made it most thought provoking are these paragraphs:

"Think about your role at work. Now assume for a moment that you no longer have any positional authority—you’re not a project leader, a department head or vice president. There’s no title on your business card and you have no direct reports. Assume further that you have no way of penalizing those who refuse to do your bidding—you can’t fire them or cut their pay. Given this, how much could you get done in your organization? How much of a leader would you be if you no longer held even a tiny, tarnished scepter of bureaucratic power?".
"... how much of your power comes from what you are (the VP for HR, for example), and how much comes from who you are ..."

In my mind, that's powerful stuff. Of course leadership is enhanced by title and power - perceived or otherwise. Who isn't going to follow someone, at least temporarily, who can dump your ass into the cold winter of this recession.

Yeah, Gary Hamel's article could be useful as an exercise in self-examination by existing leaders but I think a lot of that internal dialog has already happened. At most, this discussion could be a catalyst for calibrating oneself or possibly a reminder of some of those leadership principles. That's not what I find most interesting about this article.

I think this article is most useful for those that are not currently in an officially sanctioned leadership role. It's for self-examination of whether one is acting like the leader one thinks they are, or thinks they want to be, or possibly reluctant to be, or even in denial of being.

Another valuable aspect is to identify the natural leaders on your team and within your organization. Who is being followed without a title? Who on your team and organization tends to be at the center of things? Who do people go to for questions and advice? Those are the people that will build the teams for the future who are entrepreneurial, innovative, and mission oriented. You will need them because today's workforce is transient - and this is especially true in software engineering.

I recommend reading Gary Hamel's article How to Tell If You're a Natural Leader as well as subscribing to his Management 2.0 blog.

I'm interested in your thoughts.

Kanban Development Methodology

2009-06-10T04:24:00.000-07:00

I was introduced this week to a development methodology called Kanban. For those of you who don't know the origin of the word Kanban, it's manifestation is as a physical card as part of the Toyota Production System (TPS) that signals the moving and production of parts in a "pull" system; a system where parts are made available as others are pulled into use downstream. The objective in a manufacturing setting is to control inventory based on the speed of production. The concept is applied to software development through controlling the development of tasks. In Agile, tasks could be user stories (the parts) which are subsequently developed into software features (downstream production).

The control of requirements, design, and development inventory is important to maximize project efficiency. A main focus of Agile is to gather requirements, design solutions, develop objects, and test working code at the moment they are needed. If any of these areas are far ahead or behind the others then a critical principle of Agile is being violated. Implementing Kanban helps to keep production of objects at each stage of the project at the appropriate levels.

Here is how it works. The team has a board that looks very similar to a Scrum board. It has columns for stages of development. The stages can look quite different. The images contained in this article are a few examples courtesy of InfoQ. Some look eerily like Scrum boards, don't they?

Regardless of the column names, cards start out in the far left column and move their way to the far right column. The inventory of cards at any one category is controled by open slots within a category. For example, using the image with the multi-colored postIts, when a card is pulled from the To Do column and placed in the Doing column, it leaves an open slot in the To Do column thus a need for more To Do items to work on. Unlike Agile, Kanban does not bundle work into sprints. The flow is constant.

In my opinion, where Kanban is most effective is on larger projects where there are teams that make up "columns" of work. The image with the Waterfall style stages illustrates this best. A large project will likely have teams that make up Basic Design, Detailed Design, Development, and Validation. With Kanban, each team can set their capacity of work by the number of slots they make available. If the Basic Design team has 10 slots available then they are saying they can work on 10 basic designs at one time. Detailed Design may have only 7 slots, thus their capacity is 7 Detailed Designs at a time.

It isn't hard to imagine Agile teams using Kanban techniques to control the flow of work. A simplistic view is to wrap Kanban into iterations and you now have Kanban Agile. There are additional difference between Agile and Kanban and I provided a couple of links at the bottom of the page if you are interested in learning more.

In a previous article, The Convergence of Continuous Integration and Continuous Release, I discuss the merits of deploying to production immediately after code is checked in and all test are run and are successful. Kanban would work exceptionally well in this type of environment.

Whether it's Agile based on Kanban or another style of Agile, there is a consistent attribute of Agile teams; the Agile team needs to be composed of multi-faceted team members who are adaptive. As production velocity shifts across any or all categories of the project, team members should be able to shift into different roles accordingly, i.e. gathering requirements, designing solutions, developing software, and/or testing working code (or coding automated tests). Personally, I think this makes being a software development professional more interesting but it isn't for everyone.

Click here and here to read two good articles on Kanban as it relates to Agile software development.

I'm interested in your thoughts.

The Emergence of Enterprise Social Computing

2009-06-04T15:50:00.001-07:00

It’s exciting to think that true social computing at the enterprise level is right around the corner. That in the near future, the pretense of living intranet sites will be replaced by open, transparent, and constantly evolving intranet sites that grow organically through the contributions of the organizations workforce.

Picture, if you will, an environment where organizational leaders blog to their workforce on a regular basis and readers comment freely. Where those same organizational leaders query their workforce and their answers shape the company’s roadmap.

Add to that the ability of each member of the workforce having the ability to customize their own corporate page; blog on their respective expertise; create communities of like-minded people with whom they work regardless of proximity; search the organization for specific skills, work product, like-projects, structured and unstructured data; where they search for people with specific past experiences related to companies, industries, and expertise. All this with the ability for spontaneous contact through IM, email, VoIP, and web conferencing.

Lastly, this environment is shaped via email postings, blog articles, project and organizational processes, discussion boards, forums, wikis, profiles, and virtually any other type of medium where information is shared.

Today I attended a seminar at Microsoft in Waltham, MA where my colleague, Mauro Cardarelli, as well as other professionals in the SharePoint world, demonstrated real-life implementations of the environment described above. It was clear evidence that this type of environment is ready for prime-time and available to any organization with the vision and discipline to implement it.

The organization that is looking for a fully quantifiable ROI will be late adopters. But I’m certain they will be adopters. There are many aspects of this that are quantifiable but there are many more benefits that are not. One has to know that a fully collaborative organization is a fundamentally more productive one.

The Convergence of Continuous Integration and Continous Release

2009-05-16T05:11:00.000-07:00

There is an emerging paradigm with continuous release that may result in a convergence between it and continuous integration.

The concept of continuous release is not new. However, in most instances where the term is used it refers to deploying and testing the application on a periodic basis - usually nightly - and deploying the entire application. With the latest paradigm, continuous release becomes more like continuous integration - when changes to a component are checked into the version control system, not only does it result in an immediate action of building the component but is extended to include testing and releasing it.

With this paradigm, which I'll call true continuous release, the benefits are:

In true Agile fashion it provides immediate user feedback thus adding another layer to the Agile feedback loop.
Safety with assuming the risk of integrating smaller and more efficient releases as opposed to a much larger releases of whole applications or large subsets of it.
Users are provided the flexibility to elect which features they want to upgrade/install.

A few of the requirements for true continuous release are:

Strong feature coverage with automated testing.
Tight process with recording component versions.
Significant complexity related to the recursive nature of component dependencies.

I highly recommend a white paper on this emerging release methodology that you can find here. It contains valuable information of its benefits as well as implementation difficulties.

Is this the next efficiency model? Is its value limited to companies such as online gaming companies that are structured based on frequent and noticeable changes? Are its complexities too significant that most organizations will pass on implementing it?

I'm interested in your thoughts.

People, Processes, and Protection

2009-05-02T09:50:00.001-07:00

I'm going to drift a bit from my typical subject matter and talk about the importance of managing people and processes for the protection of all involved.

You may wonder in what context the word protection applies. Protection is the obligation of managers to protect their direct reports, direct reports obligation to protect their manager, and everyone protecting their customers and organization. The glue that holds all of the various protections together is a codified set of processes. I know codification is optional, but to be great instead of mediocre, the processes need to be as well understood and unambiguous as possible thus documentation is required.

Example:

Background: A consulting company has a custom software development effort with their best client. The project is highly visible and the client's financial investment makes it high risk as well. Each two week development cycle has a scoped set of features that the development team commits to and at the end of which results in a demo of the working code to the customer.

Manager Protecting Direct Reports: The development manager works with the development team to scope the features to be developed during the current cycle. Each team member commits to one or more deliverables and provides their estimated level of effort. The development manager is doing his/her best to set up his/her reports to succeed by involving them in the scoping, assignment, and estimation processes.

Direct Reports Protecting the Manager: As the cycle progresses, each developer is cognizant of the commitments to the client and all of them know that no one likes surprises. When a feature is bigger than expected or unforeseen obstacles arise that threaten the ability to deliver the scoped feature(s), the development manager is notified as early as possible to allow him/her to manage expectations with those outside of the team.

All Protecting the Customer and Organization: Because the development team does a good job with making the manager aware of the risks as they become known, the manager can notify the customer as early in the cycle as possible. Once the risks are known they are managed throughout the cycle. This allows the customer contact to manage expectations at their end. A byproduct of being disciplined with this process of protecting the customer results in successful projects and enhancing the reputation of the development team and organization as a whole.

Team Value System

Protection and process is all great but those items don't define what is being protected and why were the processes developed. Before processes can be designed and implemented there needs to be a team value system. The value system consists of the things the team views as important principles that are required to be successful.

There are several steps required to crystallize the team's focus of its value system as well as defining who the team is, where it's going, and how it's going to get there. Each of the steps should align with the organization, cost center, department, team, and individual.

1. Vision and Mission Statements: The team needs to know unambiguously why it exists and where it is going. If you don't have them, then write them. You'll be surprised at how much thinking you'll need to do.

2. Team Value System: A team's value system is essentially the rules of engagement at the individual level. Here is a small example:

Team First
- We constantly strive for balance of skills throughout the team.
- There is redundancy within the team, e.g. primary/secondary client contacts.
- Choosing the right resource for a particular task/project is dependent on:
  - Task/Project Context
  - Required Skills
  - Desired career path of team members
Communication
- We err on the side of over-communication
- We are collaborative with solving problems
- Conversation is more accurate than written correspondence
Customer

Everyone is everyone’s customer
Super-pleasing is the standard level of service

3. Team Composition: This is the act of documenting how your team should be assembled; essentially what is the strategy with regard to the the required skills within the team and how that manifests itself into the types or categories of people needed to most effectively perform day to day activities. I'd suggest categorizing your current team members into each of the types that you identify to see if you are aligned properly.

Protection Through Process
Once the three steps are complete it is time to create the processes that ensure the value system is efficiently implemented on a day to day basis - without significant dependency on human oversight. Heavy involvement by people to ensure the plan is executed daily is inefficient because of the transient nature of people, the periodic unavailability of people, the expense of human oversight, as well as a host of other reasons. Processes need to be developed in a way where, for the most part, the team runs itself. This paradigm is scalable. When specific people leave the organization or become unavailable for periods of time, the processes don't fall apart. It also leaves people more time for innovations that make the processes stronger.

Teams and its members have good intentions that sometimes go awry. In most cases when things don't go as well as expected it is because the team/individuals don't have a clear idea of their value system and/or poor processes that don't protect them. The idea is to consistently put people in a position to win. In my experience, implementing the tools described here help teams be the best that they can be.

I'm interested in you thoughts.

Estimating Probabilities for Delivery (< 10 Tasks)

2009-04-22T03:38:00.001-07:00

Delivering an application on the date specified at the beginning of a project is rarely, if ever, achieved; especially if the delivery is expressed as a single date (commitment) instead of a range of dates (estimate). However, there are ways to estimate the probability of delivering on specific dates using relatively simple formulas in just two steps.

Step 1: Calculate the standard deviation for each task
This isn't as scary as it sounds. Here is the formula:

StandardDeviation = (SumOfWorstCaseEstimates - SumOfBestCaseEstimates)/6

Your estimates are based on Worst, Best, and Expected cases, right? I hope so. The denominator (6) in the formula counts as one standard deviation. One standard deviation means that 99.7% of your estimates will fall within the Worst/Best range. Depending on the risks associated with the project or your skill with estimating you may want to decrease the denominator as a buffer. For example, using 2 as the denominator means that 68% of the actual effort per task will fall into the Worst/Best range. Steve McConnell's opinion in his book Software Estimation: Demystifying the Black Art is that accomplishing a 68% accuracy is achievable with practice so you might want to start out using a denominator of 2 or less until your historical data tells you otherwise.

Step 2: Calculate the probabilities for delivery
The next step is to apply the standard deviation to calculate the likelihood of delivery. Below is a table with the probabilities:

Percent Likely	Calculation
2%	Expected case - (2 x StandardDeviation)
10%	Expected case - (1.28 x StandardDeviation)
16%	Expected case - (1 x StandardDeviation)
20%	Expected case - (0.84 x StandardDeviation)
25%	Expected case - (0.67 x StandardDeviation)
30%	Expected case - (0.52 x StandardDeviation)
40%	Expected case - (0.25 x StandardDeviation)
50%	Expected case
60%	Expected case + (0.25 x StandardDeviation)
70%	Expected case + (0.52 x StandardDeviation)
75%	Expected case +(0.67 x StandardDeviation)
80%	Expected case + (0.84x StandardDeviation)
84%	Expected case + (1 x StandardDeviation)
90%	Expected case + (1.28 x StandardDeviation)
98%	Expected case +(2 x StandardDeviation)

There are more complex methods for calculating deliver dates for projects that have greater than 10 tasks. I’ll be writing about those in future articles.

I have a variation of the above in the form of a spreadsheet that I use as a template for inputting tasks with their respective estimates. The spreadsheet will automatically calculate the standard deviation and percent likely delivery dates. If you would like me to send it to you feel free to email me at jack@notarangelo.com and I’ll send it along.

Workflow Patterns

2009-04-16T04:02:00.001-07:00

A colleague of mine gave me good advice a while back about blogging. He recommended that I submit articles frequently and that they be short and concise. I'm not sure how proficient I am at writing short and concise articles but my guess is that this one will be the shortest and most concise of those that came before it or will be written after it.

I stumbled across an excellent web site for workflow patterns . The site contains research on workflow patterns conducted by various academicians and are sanctioned by the Worflow Management Coalition. Below are links to the four papers that I think are a must read if you are working with or developing any kind of process-aware software. You can view the content on the web site directly or download a PDF file at the top of each link with all the patterns under that respective category.

Control-Flow Perspective
Data Perspective
Resources Perspective
Exception Handling Perspective

Agile Feedback Loops

2009-04-02T12:24:00.000-07:00

One of the most important aspects of Agile is the feedback loop contained in virtually every aspect of the methodology and the built-in efficiencies related to it. I know I'm being Master of the Obvious, but knowing about a problem immediately and thus fixing it immediately is so much easier and manageable than untangling many problems that were unknowingly compiled over the course of months and with a short amount of time to straighten everything out. Aggghh .. it's stressful just thinking about it. Below are the places in the Agile methodology (or at least the way I have implemented it) where we see feedback loops.

Continuous Integration (CI)
When code is checked in, the application will successfully build or it will not. On my team, we get automated emails after the CI process runs regardless of success or failure so feedback is received in both positive and negative outcomes. The rule is - don't leave for the day until your checked-in code successfully builds and the unit tests succeed. When the CI process fails, it is the top priority of the team to fix it.

Nightly Build and Deploy
Every night the projects that make up the application are built and deployed to a distributed environment. During this process the compilation, automated unit tests and automated QA tests should succeed. If anyone of those processes has a failure, then, just like with the CI process, it is the top priority of the team to fix the failure.

Daily Stand-up
Every day the project team meets for 15 minutes where each member tells the their team members what they worked on yesterday, what they are working on today, and what obstacles they have. As a result, missteps in priorities is only a day away from being corrected and information about what someone else is working on that is pertinent to a developer is communicated every morning.

Sprint Retrospective
At the end of every two week sprint, the team discusses what they think went well and what went not so well. This feedback provides the necessary information to tweak our processes accordingly. Little by little our processes gets stronger.

End of Sprint Demo
This is where the rubber meets the road. At the end of every two week sprint we have an immovable demo to the client. What does our customer think about our work? Did we get it right or wrong or a combination? What insight is gained about the application as a whole as a result of this sprint's development and what is the effect on the priorities for future development?

I was in a client demo yesterday for Sprint 12 - thus it was my 12th demo for this particular project with 4 more to go - when I had a somewhat out-of-body experience. As we were nearing the end of the demo where I was peppered with questions by a room of 10+ client stakeholders asking "will it do this? will it do that" of which my response was "well let's see" (and in virtually all cases the app performed well), I mentally became detached from the demo process and saw the entire project life cycle in my minds eye and was stunned.

That's when I said to the group "Can you believe how well this is working? This is amazing." And, that's when they started making fun of me by asking me if I wanted to be carried around the room. By the way, having a demo every two weeks with your customer promotes a positive bonding experience (assuming the demos are successful).

The reason for my impulsive utterance was that after developing software for the better part of the last 15 years (a lot of which was not using Agile), I'm still stunned how well Agile works. Of course you need high quality developers too but the constant feedback loops and addressing issues immediately instead of during UAT is directly related to successfully demo-ing an application 12 times with minimal glitches.

Why Is Software Estimation Always In the Backseat?

2009-03-28T10:02:00.000-07:00

I don't understand why software estimation doesn't take a more prominent role in software development.

Virtually everyone who's livelihood is related to software development has experienced projects where team members have worked a zillion hours, the project went over budget, had features ripped out to meet a date, was delivered with poor quality, etc.

Here are some data to provide context into our experiences in relation to the software development space as a whole (thanks again to Steve McConnell's book "Software Estimation: Demystifying the Black Art"):

Many studies have concluded results that state 25% of projects will be canceled, 25% will be on time and within budget, and 50% will be late and/or over budget.
A project with 10,000 function points, which is a mid-sized project, has a 1% chance that it will be delivered early and a 20% chance that it will be canceled. As the number of function points goes up so does the failure rate.
On average, a late project is 120% late and an over budget project is 100% over budget.

It is likely that the percentages for late/over-budget is only a piece of the picture and the actual experience is much worse. When a project is 120% late and/or 100% over budget, there is severe urgency to deliver. When that occurs functionality is often stripped out of the original scope and teams are forced into a death march to get the project done. Most teams do not pay overtime so whether a developer works 8 hours or 18 hours, the effect on the budget is the same. As a result, the overage stats do not reflect that what is delivered is likely less than what was scoped and the hours applied are far more than the budget overrun indicates. It is also reasonable to assume that in that environment design time suffers causing a more difficult application to maintain, extend, and scale.

The moral of the story is that the our industry is dysfunctionally addicted to under estimating our projects! One major reason for this, believe it or not, is expert judgment. There are a lot of smart people with loads of experience in our industry and many will estimate tasks, and thus projects, based on gut instinct instead of quantifiable data. This is by far the least reliable estimation technique.

In the next article, I'll provide a few methodologies that you can use that are fairly easy to implement and will increase your likelihood of success.

Software Estimation v.1

2009-03-20T04:44:00.000-07:00

I recommend to everyone who is involved in any way with software development to read the book "Software Estimation: Demystifying the Black Art" by Steve McConnell. I laughed, I cried, it changed my life. Well maybe not the first two, but it did change the professional side of my life.

There's so much to software estimation that one article cannot do it justice; especially if the objective of the article is to keep it short and concise. This first of probably many blog entries on software estimation will be dedicated to distinguishing between an estimate and a commitment. This is a vital point because the two are often used interchangeably.

Estimating is an unbiased analytical process. With regards to software development, the objective is to approximate the amount of time and/or cost to develop software to solve a problem or satisfy a specific need. As an approximation, an estimate should be communicated as a range, e.g. "it is likely that this project will be delivered in 3-5 weeks".

A commitment is much more definitive. It is an agreement to deliver specific functionality on a specific date for a specific cost, e.g. "the project will be delivered in 6 weeks."

Related to a commitment is a target. A target is a biased process based on the goals of the business, e.g. "We need this software delivered by June 15 to demo at the premier industry convention!".

In my experience, more often than not estimates are expressed as commitments which are influenced by the business driven target.

Communicating estimates as a single point number vs as a range is misleading (in most cases unintentionally). Every software development effort is, in actuality, an invention and an invention cannot be guaranteed to be completed on a certain day for a certain cost. As we all know, our estimates are not 100% accurate. However, if we communicate estimates as a single point number we are implying 100% accuracy.

Developing high/low cost estimates is fairly common and as you can probably guess I recommend communicating both to your customers. There are many techniques to quantify, thus making credible, high/low estimates. Although estimation by intuition is one estimation technique - and is probably the most frequently used - it is the least reliable. I'll be writing about some of many techniques I garnered from Steve McConnell's book in future blog entries.

On the other hand, dates are rarely communicated as high/low estimates. Typically they are communicated as commitments (single point date) which often result in very long days for developers, negotiating with the client to deliver at a later date, removing features from scope, providing a poor quality product, or a combination of some or all of those. I have a question for you, when was the last time you presented your delivery dates as probabilities, e.g. "there is a 25% probability of us delivering the software in 11 weeks and a 98% probability that we'll deliver it in 15 weeks."? The techniques for quantifying those probability statements will come in later articles too.

I'm interested in hearing about your issues/resolutions to software estimation problems.

Creating and Maintaining a Sense of Urgency

2009-03-14T10:56:00.001-07:00

A consistent sense of urgency is one of the separators of great teams from all of the others. Urgency promotes teamwork, focus, efficiency, collaboration, pragmatism, vision, and all the other important characteristics of a highly productive team and project.

I’ve been on projects where the management team wanted desperately for our team to have a sense of urgency but was unable to create it – never mind maintain it. Assuming a project team is made up of talented people who enjoy what they do for a living then the reason for the lack of urgency falls on management’s inability to provide a conducive atmosphere to instill it.

Knowing the ingredients that creates a sense of urgency is the hard part. Actually creating urgency is simple. All it takes is a disciplined approach to process by management.

The key ingredient are:

Short and tightly focused goals that roll into medium term goals which roll into longer term goals.
Individual accountability.
Visibility into the goals of the team and it’s individual members.
Knowing the dependencies each member of the team has on one another.
Everyone’s involvement with process improvement.

Every project already knows the long-term goal – deliver the product that is mutually agreed upon between the developers and customer.

Agile does a great job of providing the short and mid-term goals via the daily stand-up and sprints, respectively. At the beginning of the sprint, the development team provides a scope for the sprint, which is usually between 2 and 4 weeks at the end of which is a demo to show what was accomplished. What rolls into the sprints are the daily stand-ups. Each member of the team provides an update on what was accomplished yesterday, what’s planned for today, and any obstacles they have.

For as far back as I can remember, my father has always emphasized that “if you take care of the little things then the big things will take care of themselves.” The daily stand-up epitomizes this philosophy. If we strive to consistently accomplish our goals on a daily basis, then it’s reasonable to assume that we should accomplish the goals for the sprint. The same holds true for the relationship between the sprints and the project as a whole.

The tightly focused goals that revolved around the daily stand-ups and the sprints creates an environment where creating and maintaining a sense of urgency is built-in. The best part is that it is self-maintaining. It doesn’t require constant reminders and direction from management. The best processes are those that work on auto-pilot where a manager’s job is to nurture it, make sure the team stays disciplined to it, and looks for ways to improve it.

That begs the question – as a manager, how should my sense of urgency be created and maintained? That’s a blog for another day.

Test Driven Development (TDD) Tip

2009-03-13T05:58:00.001-07:00

I'm a big fan of Test Driven Development (TDD). TDD is an Agile methodology where automated unit tests are developed before the feature is developed. By coding automated test before actually coding the feature, the developer is forced to think through how the feature should work. If you've ever developed manual test scenarios you know what I mean. Many questions come out of the wood works of one's mind when going through this process. Another advantage of TDD is that once the tests are developed the development mission become clear – code to the tests and once they all succeed you are done and ready to move on to the next item in the prioritized list of features.

However, TDD requires a bit of procedure built into it – especially with regards to Continuous Integration (CI) and deploying working code on a nightly basis. Both processes are vital to the success of a project. I encourage my team to check in working code daily; ideally multiple times per day. With each check-in the CI server builds the project(s). If the project fails to compile or the unit tests fail then the build is considered failed. At that point, everyone on the team gets an email stating the failure and fixing it become the top priority. This also applies to the nightly deploy process which compiles the code, runs the tests, deploys the project into a distributed environment where automated QA scripts are run. A failure at any point results in top priority work for the team the next day.

In case you haven't already deduced the problem, if a feature takes a week to develop and all the automated tests are written up front, then the CI and deployment processes will be in failure mode most of the time. Thus, there are three solutions:

Iteratively develop the tests. For example, in a single day a developer may create a few classes, some properties, and a few methods. Tests should be written to accommodate what will be completed today not for the entire feature (this is my preference).
The developer could develop all the tests with an "ignore" flag so the tests don't run.
Use a methodology other than TDD.

I'm interested in your thoughts and alternative approaches that you've seen.

The Miracle of Agile

2009-01-17T07:00:00.000-08:00

I am a disciple of Agile as a software development methodology for the simple reason that it was brought to Engineering teams through divine intervention. That statement might contain a bit of hyperbole but only a bit. Agile has revolutionized the way in which software is being developed by many teams across many industries though it is still unknown or a mystery to many more.

What is Agile? Agile is an iterative approach to software engineering who's precepts are: collaborative teams consisting of cross functional members; frequent validation of requirements, designs, and implementations; self organized teams; and, unambiguous individual accountability.

How is Agile different?
I'm sure everyone reading this has heard of processes such as requirements gathering, analysis and design, development, QA testing, and user acceptance testing. Agile as well as many of the tradition approaches, Waterfall being by far the most common, embrace these principles. The differences lie in the implementation of those principles. Where Waterfall will attempt to describe the the entire application upfront through documentation which is then followed by development, QA testing, and user acceptance testing, Agile bundles those principles into iterative life cycles called "sprints". Sprints are of a fixed duration, commonly between 2-4 weeks, and repeated over and over again until all the features are developed, the project budget is exhausted, or time runs out.

Plan-Driven (Waterfall) vs Value-Driven (Agile)
The way in which the principles are implemented is the manifestation of their difference in philosophy. Where waterfall is plan-driven, Agile is value-driven. A plan-driven methodology is heavy on documentation and strives to fully describe the application before developing it through requirements, analysis, and design. That process provides a framework to methodically develop what's been documented. Typically there is a gargantuan project plan that is associated with the effort and everyone marches to the plan with little regard for course corrections and re-evaluation.

A value-driven methodology stresses an empirical engineering process using an inspect and adapt approach with frequent feedback loops, i.e. sprints. The reason for this is that Agile believes that requirements gathering, analysis, design, development and testing should happen together when the feature is ready to be implemented. This approach provides flexibility with project changes such as feature deprecation, adding new features, and changing the requirements of existing features. As these events occur in a Waterfall project the documentation becomes cumbersome to maintain. The most efficient time to write about it is when it's time to develop it. During an Agile project, it's easy to make course corrections such as changing requirements as a result of what's been developed previously or a reconfiguring development priorities because of extenuating circumstances.

Although it's comforting to know that so much thought went into determining what the customer wants and how their requirements should be implemented, there are significant inefficiencies with the level of detail that goes into the upfront analysis. The reasons are simple, some of which are:

Users aren't always clear in their own minds what they want and how they want it.
Large documents are not handled well by many people. They tend to be too abstract for people to fully grasp.
Things inevitably change in the minds of many users once they can see and feel their requirements implemented.
The business environment often changes and directly impacts the priorities of the project.

In my mind, it makes perfect sense to get your customer to see and feel the application features as soon as possible. This doesn't mean abandoning requirements, analysis, and design. It just means that upfront planning should be limited to:

Project Initiation: Planning the sprints, project infrastructure, team members, communication matrix, etc.
Developing a Prioritized List of Features: This is a list of features, prioritized by importance, with user-stories on how the features will be used.
High Level Architecture: A high level architecture should be developed. This phase determines things like the application tiers, technologies employed, thick or thin client, etc.

Once those steps are taken then development should start. I prefer two week sprints at the end of which my team gets immediate feedback from the customer. Catching mistakes with interpreting the user requirements, or users realizing that their requirements need to change is far cheaper to address at this point in development than at the end of the project during user acceptance testing. The recurring customer demonstrations also builds user confidence because they see progress. From the developer perspective, sprints provide a clear mission and the immovable customer demo at the end provides a constant sense of urgency.

Why isn't everyone doing it?
There isn't a single answer as to why everyone isn't using Agile but the most common is - you guessed it - resistance to change. Here are a few reasons for resistance:

The Devil You Know: People are comfortable with what they know and uncomfortable with what they don't know. In my experience, once the change is made, most team members see more similarities to their previous approach than they thought were there otherwise. The apprehension of the change is greater than the actual change itself.
Personal Fear: Can I do this? Will I like it? Will I learn it fast enough? Will I look foolish? How will my job be affected?
Risk Aversion: The chances of switching methodologies without feeling some amount of pain is low. It will take at least one, but probably several, projects to change behaviors where Agile processes are implemented and feel natural. The speed of the transition is dependent on the fervency and commitment of the transition evangelist.
Existing Team(s): Agile is highly collaborative and tends to minimize attention on documentation, thus an inherent lack of detailed specifications. If your projects rely heavily on coding to specs, then the transition to Agile could be challenging. However, if your team is already working from prototypes, has close involvement by the customer, and is highly communicative, then you are already working in an Agile-like style and the transition will likely provide change in the form structure not in approach.
The Iron Triangle: There is contention with each side of the iron triangle - features, cost, and schedule - and it is virtually impossible to fix commitments to all of them simultaneously. Agile makes clear that commitments can only be applied to a maximum of two of the sides. One side always needs to be fluid. For example, company x has a demo at a convention in 3 months where it must show y set of features, therefore the the schedule and scope are fixed so the cost needs to be variable. With Waterfall projects the commitments to date, scope, and budget are typically, and irrationally, fixed at the beginning of a project. This makes people feel comfortable. However, in reality one or more of those commitments will inevitably be violated because it is unlikely that enough is known to allow for committing to all three.

Although it is unlikely that Agile was brought to us through divine intervention, the genius in it's approach makes me wonder what took so long to get here.