Nov 13, 2008 / Software Development is Complex Adaptive System. No Doubt.

1 comments

Complexity science is very young. Cybernetics, ecology, sociology, meteorology all study complexity. In general, a complex system consists of interacting components. The result of this interaction can't be predicted by observing an individual component. For example, human brain consists of neurons. The brain has 'memory', while each neuron doesn't. Ants’ colony behavior can’t be predicted from the behavior of an individual ant. And it is impossible to understand software development process looking at how just one developer works.

We are especially interested in a particular case of complex systems called Complex Adaptive Systems (CAS). The main difference is that CAS may learn and change (adapt) over time based on previous experience. CAS remember the history, and that is the main difference from chaotic systems.

There is no common definition of Complex Adaptive System (CAS). One of the most popular definition was offered by John H. Holland

CAS has quite many common properties. However, hardly a system should have all of them to be CAS. Moreover, researches define different sets of CAS properties. Let's try to review the most common properties and see how software development process can be described in terms of those properties.

CAS Property	Software Development interpretation
Agent System component.	People in development team and others involved (product owners, stakeholders).
Feedback Each agent in CAS reacts to information flows. In return, the information that this agent will receive in the future is influenced by his actions. This feedback may strengthen or weaken agent actions.	Development team relies on previous experience (previous project, previous iteration) for decision making. Decisions influence future actions.
Emergence Emergence is the way complex systems and patterns arise out of a multiplicity of relatively simple interactions. Small actions of agents lead to unexpected emergent system behavior and it is impossible to predict system behavior based on the behavior of an individual agent.	Small errors pile up and may cause huge problems, even stop project at all. Each person in a development team does not have all info about the project. People communication and collaboration leads to unexpected results.
Non linearity Feedback and emergence cause non-linear system behavior.	Will a project success or fail? You never know from the start. Predictability in software development is hard (almost impossible in many cases). So, you can't create a big plan upfront, you can't create a big design upfront. Non linearity is a natural property of CAS, so all that BDUF, BPUF are just plain wrong attempts.
Sensitivity to initial conditions Also known as Butterfly Effect. . Even one small change may lead to significant effect. Again it plays against predictability, you can't predict weather for more than 2 weeks. Butterfly effect is a property of chaotic systems, so CAS is a chaotic system as well.	A small bug in the code may cause huge problems to the end user. Unavailability of a key project person on some particular day may cause major slippage because a wrong decision was made while this person was away. I bet you have such examples from real life.
High Adaptability CAS is very flexible and adaptable. It can keep its basic structures and behavior, reacting to changes in external environment and taking advantages of these changes.	Development team should react to environment changes (new requirement, technology change, lead developer loss, etc). It should adapt to the new conditions and try to take advantage even from the worst events.
Information exchange Agents exchange information.	People communicate in development team and outside development team (stakeholders, product owners, etc).
Cooperation Agents in CAS cooperate to reach the goal.	People in development team cooperate to produce the software. BTW, game theory studies cooperation and may be applied to software development. That was already done by Alistair Cockburn.
Specialization Agents may have specialties. Strong CASs have a number of various agent types. This provides more opportunities to use environment features and to adapt to them.	Obviously, we have Developers, Testers, Scrum Masters, Product owners, System Administrators, etc. If a team consists of developers only, it will be less efficient than cross-functional team.
Self-organization There's no hierarchy of command and control in a complex adaptive system. There's no planning or managing, but there's a constant re-organizing to find the best fit to the environment. The system is continually self-organizing through the process of emergence and feedback.	Traditionally, there is hierarchy in software development. There is a Project Manager who rules the project, there are Functional Managers, there is VP of Development, etc. In agile development there is no hierarchy. The goal is to have self-organizing team.
Edge of Chaos A system in equilibrium does not have the internal dynamics that enables it to respond to the environment and will slowly (or quickly) die. A system in chaos stops functioning as a system. The most productive state to be in is at the edge of chaos where there is maximum variety and creativity, leading to new possibilities.	It is hard to create software in Order. Development team will not have enough creativity to resolve unexpected and complex problems. That's why heavy methodologies do not fit software development. It is impossible to create software in Chaos. We all know which results hack & fix brings. Ideally we should produce software using a relatively light methodology (agile?). A set of simple rules allows edge of chaos and powers creativity, flexibility and success.

This is just a quick overview of common CAS properties. Later we will go into details and draw more parallels between CAS and software development. But even in this short list we've got some nice phrases like self-organization, feedback, information exchange, cooperation, adaptation - does it remind you something?

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development

Labels: complex adaptive systems, software development

posted by Michael Dubakov @ 3:08 AM

Nov 12, 2008 / Agile Software Development and Complex Adaptive Systems: Intro

3 comments

There are quite many agile processes in the world already. We have Extreme Programming, Scrum, Crystal, DSDM, Feature Driven Development, Lean. Some of them are famous, some are maybe dying. The point is, we have plenty of agile processes. In general this is not too bad. We've got alternatives, they move agile forward and solve real problems. On the other hand, I feel the lack of information on the roots of agile processes. Most articles available are missing the point on my opinion.

Why iterative development works? Why agile manifesto is the truth? Why traditional project management techniques miserably fail on software development projects? Somehow many people feel the right direction. They have a background that guides their decisions. We feel that agile manifesto brings real value. We has proven this in practice after all! But I personally feel that there is something deep and cool behind most of agile practices. Something that will explain why this all works.

While reading Jeff's post about Scrum roots, I found a reference to Complex Adaptive Systems. I went digging into more details. I read a lot about CAS and related things and there were many "a-ha" moments. There are so many parallels between CAS and Software Development. It is pretty obvious that Software development is a Complex Adaptive System itself. It has almost all basic properties of CAS and it is interesting to read, for example, about adaptation in various CAS and think about its application to the software development and agile principles.

I believe that a deep understanding of Complex Adaptive Systems will shed some light to the real roots of agile development and will answer important questions. With this knowledge in hand it will be easier to see why this works and why that failed. It will be easier to make decisions and create a better software faster.

However, I am not sure how interesting this will be to the people :) In the next several posts I will write on why software development is CAS, describe important properties of CAS and apply them to software development (adaptation, feedback, self-organization, emergence, edge of chaos, non linearity, etc). Let's see how it goes, maybe we'll find something interesting ;)

• Part 1. Agile Software Development and Complex Adaptive Systems: Intro
• Part 2. Software Development is Complex Adaptive System. No Doubt
• Part 3. Edge of Chaos and Hyper Productive Software Development Teams
• Part 4. Simple Rules, Complex Systems and Software Development

Labels: agile, complex adaptive systems, software development

posted by Michael Dubakov @ 12:55 PM

Sep 11, 2008 / Software Development Diseases and Agile Medicine

3 comments

It is really interesting to check some established diseases in software development and investigate how they can be resolved using agile processes. If agile development is so good, it should provide solutions to common problems. Here we go.

Student Syndrome

Many people will start to fully apply themselves to a task just at the last possible moment before a deadline.

Maybe the most famous disease in software development teams. You have 40 hrs task and thoughts like "Huh, one week! I am pretty sure I will complete this task in 2 days!" You are doing something fun like refactoring or research and start working on the task Wednesday's afternoon. Suddenly you encounter a serious problem. You spent all day hunting for resolution with no luck. In a weekly status meeting you say that task was not done and 80% ready, but there is completely unexpected nasty problem that popped up and broke the plans.

Looks familiar? Yes, it is. Agile development solves the problem with several simple practices:

1. Daily Meetings. You have to say something on a daily meeting. If you committed to the task on Monday and spent all Monday researching something completely unrelated you will have bad feeling during Stand-Up. It pushes you to actually start working on task. Also serious problems will not be under the carpet for days, they will be revealed and expressed.
2. Short Iterations. In short iterations every single task is important. If you have 1 week iteration you are feeling that deadline is close enough and you have no spare time. If you have 6 months project deadline in waterfall process you have an eternity. You can do wrong things for weeks till the reality hits you in the neck.
3. Short Tasks. Two days is a good maximum for a single task. It is quite hard to convince yourself that you can start coding tomorrow if you have only two days to complete the task.

Parkinson's Law

Work expands so as to fill the time available for its completion.

Indeed it is a matter of perfection and laziness. Some people may complete the task earlier and have spare time for other things. Some people will complete the task earlier and polish the solution till it shines. Some people will digress from the task and complete it on time. All attitudes may be dangerous. The result is that a whole project delayed.

Agile approach has some medicine for this disease as well:

1. Definition of Done. The task is done when it passes all assertions in DoD. The goal is to have clear, unambiguous definition. People should know when it is ready. The phrase "it will be done when it's done" should have clear meaning behind.
2. Iteration Goals. Clear iteration goals help people to focus on getting things done.
3. Short Tasks. Again short tasks may help here. In short task it harder (but still possible) to add long safety buffer.

Conway's Law

Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.

Maybe it looks quite weird, but this law has great application in software development. If you have two modules that interact via API, you hardly create good API unless developers of the both modules communicate together. It leads to some conclusions:

• Communication between developers is very important for good design.
• Distributed teams may hurt software design in case of poor communication (and it is harder in distributed teams).
• Software can't be produced in isolation. You can't have several people working on several modules with no or little interactions.

In agile development there is a clear statement that communication is very important. It does not have explicit practices for software design communication (well, except pair programming), but it doesn't take a rocket scientist to create and apply them. Company should adapt and change to have great communication flows.

Brooks' Law

Adding manpower to a late software project makes it later.

There are two simple reasons behind this law:

1. It takes some time for the people added to a project to become productive.
2. Communication overheads increase as the number of people increase.

From the first sight agile development can't to anything with these problems. However that is not true.

1. Pair programming. Likely it is the most efficient way to familiarize new people with the project.
2. Iterative Development. New iteration and new release is a new mini-project in fact. It is easier for people to start with something new. If you join the team in the middle of one year release you feel less comfortable.
3. Small teams. Agile approach works best for small teams. When it scales, it does not bring much communication overhead. For example, a single scrum team works autonomously, while scrum master participates in scrum-of-scrum meetings. However it does not increase communication time in a single scrum team.

Feel free to continue the list of diseases ;)

Labels: agile, software development

posted by Michael Dubakov @ 2:32 AM

Sep 9, 2008 / Reveal Agile Development Process Problems with Kanban Board

0 comments

In previous post I described Kanban and its application in agile software development. Kanban board is not just a way to communicate story state, it is a tool that can reveal some problems in the development process.

There are some simple patterns that are very intuitive and can be applied to any development process. You should look for gaps on the kanban board. If you have many stories in a column before the gap, you definitely have a problem in development process.

For example, we have several user stories in Implemented column, but none in In Testing column. It means that testers are unable to process user stories. They are overburdened, or unexperienced, or lazy, or travel to Bahamas, or anything else. Kanban board can't tell what the problem is, but it shows that it is exist.

The gap may unite several columns. For example, we have some user stories in In Progress state and some in Done state. It means developers are unable to feed testers with implemented user stories. Maybe we have too many testers or not enough developers. Again it is not clear, but the problem is there.

Last case is when we have user stories in all states except Done. It is more complex case and it depends on context. In the beginning of the iteration it is definitely OK to have Done column empty, but in the halfway you expect to have something done. Otherwise testers will be overburdened in the end, we'll drop some stories, or we'll complete them in a hurry with poor quality.

As you see it is a very simple but effective way to identify problems in agile software development process. Nice side effect of the pull system!

Labels: kanban, problems, pull system, software development

posted by Michael Dubakov @ 4:32 AM

Jun 11, 2008 / Software Development, Problems and Framework Solutions

0 comments

Solving problems! Most of us would agree that is the ultimate goal of any software. It is especially difficult to create a tool that resolves several large problems. For example, effective project management is a huge problem that may be split into several smaller ones such as: easy planning, effective tracking, quality assurance, etc. What makes it extremely difficult is companies’ diversity. There are a lot of businesses with unmatched variety of processes, goals, people, cultures, tools and environments.

OK, we at TargetProcess have a niche, by supporting companies on "agile" road. However, we still face too many differentiators that need to be satisfied.

Some people want Gantt charts and Start/End dates for their tasks. To such requests we have been consistently saying “No”. Others want better high level planning, improved time tracking, easier UI, – the list is endless. We can't say “No” to all of them, since many of such requests are quite reasonable and most importantly - solve real-life problems. However, we can not add requests blindly into the Product. If we did, we would create a "bloatware scary monster" and people would start running from it.

What is the solution? The solution is to provide frameworks for solving problems. Sounds weird, but here are some examples to support this point.

Example 1.

Requests: "We badly need a report that includes all features, stories and bugs for current release", "I want to have a quality report that shows count of passed/failed/not run test cases for each user story in current iteration", etc.

You get the idea - people want and endless array of reports. One solution is to start implementing these reports as they coming — one by one. We feel the better yet solution is to simply allow people to build such reports without our help. Powerful reports engine resolves these types of concerns in a single hit. Moreover, people are creative and once familiar with the approach - will start to produce very useful reports that address all of their needs. The key is to make them feel in command and provide necessary tools!

Example 2.

Requests: "We want to group features by module! How come you don't have modules in a project?", "We have several teams in a project and want to plan and track them separately", "Your software really sucks! I can't even find all the bugs and user stories related to Safari easily!", "I played a little more with your software and I can't use it without relations between entities."

Did you get the problem? It is not an easy guess. They all look like different problems. Straightforward solution may include:

• Module entity in a project.
• Multiple teams concept implementation, including changes in relations, reports, etc. (will complicate product a lot).
• Custom fields for several entities at once.
• Dependencies between entities.

Wow, it will take time to implement all of the above features. Scary part is that these requests are just a tip of an iceberg. After implementation we will more likely receive even more "related" requests. Is there any magic bullet that hits all the targets in one shot?

Yes, there is! OK, all the requests are about categorization and relations. Categorize by module, categorize by team, categorize by browser, see all related entities that have the same value in a defined category. And the real solution is to provide an easy and flexible categorization. The answer is... tags and bundles! Let's see how all the problems above may be resolved with tags.

Problem	Solution
We want to group features by module, why you don't have modules in a project	Add Modules bundle Add required modules as tags into this bundle Set required tags when adding feature See all features grouped by tags in Tags area
We have several teams in a project and want to plan and track them separately	Add Teams bundle Add required teams as tags into this bundle Assign user stories and bugs to teams in Tags area Track progress in custom reports filtering by team tag
Your software really sucks! I can't even find all bugs and user stories related to Safari easily!	Add Browser bundle Add required browsers as tags into this bundle Set required tags when adding bugs and user stories Search by Safari tag and see all user stories and bugs with tagged with Safari
I played a little more with your software and I can't use it without relations between entities.	Search by required tag to see all related entities.

Common solutions for different problems. Tags and Bundles is a framework that will resolve almost all categorization problems. And be sure that people will find creative ways to solve other problems as well. If it is possible to plan with tags, I can't even imagine what tags will be used for (in a good way).

We at TargetProcess are trying to provide frameworks rather than straight-solution-to-one-problem. The latter approach works exclusively for the stand alone problems that cannot be grouped with similar issues.

Generalization mindset is a must in Product development, but beware premature generalization.

Labels: problems, reports, software development, solutions, tags

posted by Michael Dubakov @ 7:35 AM