Saturday, January 22, 2011

Change and Complexity: The Software Engineering Response

The central challenges for software engineering are complexity and change. Software engineering’s response to these challenges comes in two parts: breaking down the problem of building a software system into smaller, more manageable ‘chunks’ to confront complexity; and setting regular checkpoints during the process of building a software system to address the effects of change.

The breaking down results in workflows and the checkpointing leads to phases; together they constitute the software development life cycle (SDLC). The SDLC lies at the heart of software engineering.

Workflows represent sets of activities starting from understanding what users want from a software system (Requirements), to translating the language of the problem into the language of the solution (Analysis), to expressing the solution constructs in the language of development (Design), to building the system using programming resources (Implementation), and finally, verifying whether the system matches the stated requirements (Testing).

Phases, on the other hand, are focused towards monitoring and managing change. During Inception we ask, what do the users want from the system? During Elaboration, we are interested in knowing if the system is feasible. Next comes the question: How do we build the system? This is the concern of Construction. Finally, during Transition, we enquire, how do we transfer the system from the developer domain to the user domain? In a particular development life cycle, we may not know the answers to these questions when we ask them. But based on our experience and understanding, we have an expectation of what the answers are likely to be. When expectations are not met, it serves as a reality check: A change, not budgeted for, must have occurred. This makes us aware of the need to find out what changed and what all that change might affect.

On the face of it, software engineering’s response to the problems of change and complexity seems adequate. But it admits certain challenges.

An element of linearity is implicit in the way software engineering seeks to address change and complexity. Customers come with requirements, which are analysed, followed by the design of the system, its implementation and testing. If the right questions are asked at each point, Inception, Elaboration, Construction, and Transition seemingly follow one another in harmony.

But reality is much messier. Answers are seldom ready when questions are asked, customers and users change their minds all the time, and technology and business environments change. In the real-world of software development, it becomes imperative to go back and forth across workflows and phases several times, driven by a variety of reasons. Life is inherently non-linear, and software engineering is no exception. But just as in life, in software engineering too, we build our case on assumptions of linearity. And then hope to tackle non-linearity on a case-to-case basis.

The key challenge with software engineering’s response to change and complexity boils down to being able to monitor, control, and leverage the myriad feedback paths that exist in the real-world software development life cycle.

Feedback is one of the most fundamental techniques of engineering. In simple terms, feedback involves controlling an activity by regulating the input based on the output. Feedback exists at many levels, practical as well as perceptual. For example, an exception handler is a simple feedback loop. It monitors the execution of a piece of code and takes appropriate action if the outcome is not as expected. On the other hand, modifying a system’s design based on user response is also an instance of feedback. In software engineering, the difficulty usually lies in integrating feedback of different forms and at various levels into a consistent and repeatable development model.

This is the central challenge with software engineering’s responses to the problems of change and complexity.


Deepthi Ravisankar said...

Deepthi Ravisankar(MT2010031):
Design is difficult to predict and is expensive as it requires people with innovative skills whereas construction is easier to predict.
The non-linearity in real life is partly because software development is a design activity, and therefore hard to plan and cost. Also its basic ingredients are rapidly changing.And then the whole process depends on individuals involved, and individuals are hard to predict and quantify.

The key to feedback mechanism is to frequently produce working versions of the final system that have a subset of all required functionality. These working systems have limited shortcomings. They should be fully integrated and as carefully tested as a final deliverable system.

Ashish Tergaonkar (MT2010159) said...

I feel that the changes and resulting complexity are inherent part of any software which gives the user 'exciting' experience (Here 'exciting' refers to classification of requirements refer: The QFD handbook By Jack B. ReVelle, John W. Moran, Charles Aaron Cox page no. 174)
Hence it is the software engineer's responsibility to incorporate the changes which can occur in the middle of SDLC. The classical waterfall model fails in this aspect as it can have blocking stages.

According to me, 'the scalable design' is the key to counter the scale-up with growing requirements & change requests in the software.
The design should be 'model oriented' which identifies the stable parts. Also the classical principle of 'High cohesion, low coupling' should be taken into account for designing the modules.

Incremental SDLC model will also help in this scenario which can incorporate 'scalable design' & make the system more flexible.

Philip Joseph said...

The user views and requirement can change over time. Feedback is indeed helpful towards conveying the developer about the added needs.To include these changes in system, development model need to be adaptable. Spiral model or Iterative and incremental model are acceptable in this scenario. Once the requirements are available in detail the first prototype of the system can be made available to the user. Based on feedback further revisions can be made. Analysis, design, implementation and testing can be iteratively done until a stable and acceptable version is devised.


Nilesh said...

Nilesh Inamdar
Software engineering plays a critical role in dealing with change and complexity. Over the years it has been seen that user requirement keep on changing and will continue to change. The product must be made extensible. We must be able to modify the existing software with minimum amount of change without hampering(or changing) other parts. To achieve this, each and every design steps must be planned and noted in a standard manner. Standard SDLC must be followed. Regular feedback is must to avoid lot of unnecessary work. Expert opinion is very crucial in the field of software engineering as they have worked upon many projects and know better which things must be handled carefully.

shonali4u said...
This comment has been removed by the author.
ravi_endla said...

The SDLC plays an important role while developing a product.There exists simple models like water fall model and complex models like iterative ,prototyping and RAD models.But one cannot decide a unified model for all the softwares.

Choosing the model depends on what kind of software we are building.For a simple ,low budget project it is not recommended to go for complex models ,as they consume more time ,labor and cost.

On the other hand ,building some softwares in one go is not possible.The requirements are specified once some part of output seen.In such cases a slightly complex model called iterative model can be used.

Similarly there exists a bunch of models each of which can be used for solving a kind of issue/issues and selection of the model becomes a precursor for developing any software.

KRISHNA KANT HARKUT(MT2010064) said...

Krishna Kant Harkut(MT2010064):
I feel that SDLC(SOftware Development Life Cycle) must be strictly followed for all types of project development(small scale or large scale projects) as what ever we develop it is for the users and we would like to give the user the product they visualize.This is achieved by SDLC which tailors yours development according to the user's need.
Also the SDLC for any project development never ends because user's needs are not constant.Based on the user's new requirements the SDLC phases are again carried out.So it is the software engineer's duty to carry out the SDLC phases from time to time.

bhargav said...

Bhargav Naik(MT2010083)

The change in the user requirements can dealt with by creating prototype versions and taking feedback from the customers, which may reduce considerable rework.

bhargav said...

Bhargav Naik (MT2010083)

The change in user requirements can be dealt with by creating prototype versions and taking feedback from the customers, which may reduce a lot of rework in the process.

Pratibha said...

As you have mentioned, change is one of the central challenges in SE. This 'change' comes from the fact that

1. No person is clear about a product's requirements before building it. It is a never ending loop of change.

2. Perception of these requirements. There is no clear formula / rule which translates a requirement from colloquial language to a mathematical form (which would leave no room for different interpretations). Since, a person has to rely on natural language to make sense of these requirements, change is inevitable.

3. Technology is changing everyday. Today technology 'a' is in vogue, tomorrow no person will even remember it. Hence, there is a constant need to update our knowledge and fit into the picture with appropriate technology.

As you have mentioned, 'change and complexity boils down to being able to monitor, control, and leverage the myriad feedback paths', this does make software engineering processes easy to understand. This is way to converge to a space where everything is modeled as a rule and sticking to it will make everyone's life easy.

Feedback also is a cause for constant change in the requirements. If there was no scope for feedback, there would be no changes. But no changes in the requirements would be in a parallel world where everything is understood in the first requirement analysis meeting.

Vinaya M S said...

I hope the changes that the user need can be dealt till some point in time. There can be some expectation of a user which urge the software to be changed dramatically. In such cases the complexity for the development of the system/software will be increased. So can there be a limit for the expectation of change by the user at certain point in development phase we must think of.

Parvathy S.Pillai said...

Life is unpredictable. The so called 'normal course' is almost undefined when life is concerned.
Life is complex. It is not just ourselves that we have to look into, but as social beings, a lot depends on the impact each of our decisions has on others concerned with it.

The same is the case of Software Engineering. We can never be sure of what is next in store for us in the course of development of a product. It is complex as we have to cater to multiple stakeholders interests.

The 'normal course' of a linear development plan never can become a reality. We have to constantly address the changing requirements at all stages and phases.Since, as in life we learn from our mistakes and our failures are the stepping stones to future success, this has to be inculcated into software engineering process as well in the form of feedback.

In my opinion, feedback has to be present, but present right from the inception stage. The feedback to a prototype may need something to be developed from scratch. So we should not just keep postponing feedbacks to later stages of development. This can realize a lot of cost cutting and better user satisfaction.The right design is the key to avoid a lot of reworking and for this our base, which we build from study of the background, technology and analysis of user requirements, has to be made strong by frequent iterations.

Ashakiran said...

Asha Kiran B(MT2010017)
The idea of having a prototype done is not practical all the time. Also, the user's ability to comprehend prototypes could impact the very process of building a prototype. Also,is'nt a prototype something meant to be thrown away? It is a strict norm that no re-use has to happen using a prototype.
The root cause of all of this is the ambiguity in the way requirements are specified. Isn't it possible at all to come with a 'non-ambigiuos' se of requirements? May be, the world should have software engineers of specific domains? Isn't it wiser to have software engineers as part of the companies themselves; that come over to software giants to get software made? Or, are the possibilities so limited that it is impossible and unprofitable to have domain-specific software engineers? Or,are we ( software engineers )so annoyingly costly to have within non-software companies?!

Gajendra (MT2010040) said...

According to me in the each and every product development they are strictly following the SDLC model based methods since it is basically the they will apply the SDLC methods based on the user specification and this architecture or models are not static since each can design of his own model for his own product development. Since customer requirement is going to change day by day and demand for more sophisticated and extensible softwares are in demand in the market so a software engineer should think of development and design perspective that "if we design and develop a system how many years it can stay in the current changing market" in this perspective all the complexity and change related aspects to be considered.

Skanda Kumar (MT2010143) said...

The purpose of breaking down the problem at hand is to simplify the process so that it is easier to tackle the smaller components than single large one. When the problem is broken down into too many smaller components it becomes difficult to manage and co-ordinate the sub-problems. So it is better to first have bird's eye view of the entire project and the divide the problem with a proper trade-off which gives both simplicity and manageability.

Sreenidhi said...

Sreenidhi T (MT2010146):

Change is inevitable, probably the only entity that does not undergo change. Having stated that, it is only acceptable as a customer or an end user to view upon the software being developed as a customizable product, meeting his or her changing requirements.

Hence managing this continuous process of change in the development of any system, more so in the field of software engineering, is essential as an inherent quality in the developer as well as the development lifecycle.