What is the ideal world, not an easy question and I am sure that I cannot give an answer that makes everyone happy, but from my perspective, when I look at SW development, the ideal world is when you have a self organizing teams, teams who have a great understanding of what the customer want them to implement, which is not easy either...
The challenge we often face - as I see it is this:
1: How does the team understand what the customer want?
2: How do we get self organizing teams?
1. Understand the customer
Let's start with having a team who understand the customer and what they want, I recently read the biography of Steven Jobs by Walter Isaacson (fantastic book btw), and it is pretty clear that the customer does not know what they want, especially not if it is something really new.
Most teams that I have worked with, have an interaction designer in the team to help them, that does however not secure that the team create solutions that the customer want, even if the interaction designer know what the customer want, the team will most likely not interpret the design in a way that is completely in line with how the customer want it – and this is where Scrum works really well (and most other Agile disciplines), we deliver often and we are very flexible, so we can test our software with the customers frequently – and make sure that you have the entire team with you when you let the customer (or someone who represent the customer, i.e. in the mobile industry, it close to impossible to reach all customers), the benefit from having the team in the customer test (and not only the interaction designer, which I sometimes see) is that the team interpret the design and in most teams they also add stories to the backlog, so it is super important that they have a good understanding of how the customer think!
There are a lot more the team can do to be better at understanding the customer, it is all about interaction with the customer, the simple test that I talk about above is mostly a UX test to secure that we remove the worst usability issues, however this does not give the understanding of how the user think and use the software, it is often isolated to a limited test, another way to improve the teams understanding of the customer is to let the customer play around with your software and software from competitors while the team observe, we are planing a trip to the local café to get ideas and increase our understanding, we will buy coffee to those who are willing to let us observe how they use the software … ;-)
2. Self organizing teams
How do you create self organizing teams – well it is contradictory to build self organizing teams...
But what you can do is to create the framework and help the team to succeed... My experience with self organizing teams is that they are fantastic at collaboration, they are empowered, all decisions must be owned by the team, they care about each other and trust each other, the team is static, though it does work if one is leaving for parent leave (or similar) and return, but it does not work if you split the team and build other teams with the members from the first team.
One of the things that I have learned about successful teams is that they like each other or at least respect each other very much, so I personally always talk with all potential members of a team before creating the team, it is my experience that you cannot have 2 or more in the group who are not trusting or respecting each other, they might be able to work together, but you will never get a self organizing team, you will at most get a highly skilled group, which most managers are happy to get, but it is most likely never going to be a high performing self organizing team.
One thing that has proven very successful in helping teams to become great teams is appreciations – giving appreciations to each other continuously is very important in building great relations, it is important that you don't save the appreciations to a retrospective or temperature reading (feedback sessions).
Another very important characteristic with a great team is their ability to listen to each other, they must be able to listen both to professional and personal issues and not judge, allow others to make mistakes is important, I have been working with many teams and I have never seen a great team who did not get involved somehow outside work, it does not mean that they have to meet and socialize, but that they can talk about part of their life is important in building the relationship that is so important.
Can great self organizing teams build and maintain without external influence? Yes, I believe that they can, I think that most of us have seen it happen outside work, i.e. in groups of friends, specially if people are free to make the groups and have a common goal – but at work, this can become slightly tricky and I believe that management can help a lot to create the framework and to coach the team and they can also destroy the team if they don't pay attention, when focus is to maximize the short term profit, rather than building long lasting teams...