The Most Agile Thing You Can Do Is Throw Away Agile
In my career, I've found that there are two types of "agile" software process maps. There is Agile, with a big A, that is usually purchased from non-technical people by non-technical stakeholders. Big-A Agile is a series of meetings, metrics, and reports used to structure and monitor software teams. I've even experienced story estimates as hours, and a requirement to list "actuals" on each story to verify the estimate was correct.
The other type of "agile" is essentially a few pieces of advice for teams of developers looking to build software correctly — a mindset for teams to attempt to think from to create working software for stakeholders. In little-a agile, there are very few required meetings, a handful of meaningful metrics, and the strongest metric for success being a happy stakeholder.
I believe wholeheartedly that Big-A Agile is not agile software development at all. Big-A Agile is harmful to our industry.
So why would anyone pick Big-A Agile to begin with?
The companies/groups that choose this methodology are often larger companies attempting to maintain order. Imagine a product being built by 15 teams of ~10 software devs. It's a lot of people to manage, and a lot of variance to account for. The people purchasing Big A Agile software methodologies are looking to bring up their bottom lines, to ensure even their lowest producers are able to be productive. Why is this a problem, you ask? Well, if by chance any of the devs on those teams are of average or above proficiency, they're being tremendously hampered by the existence of the process. They don't have room to run, so to speak.
But How Do You Handle Scale?
Big A Agile claims to handle scale via hard process and measurable data points. The little a agile I'm describing may struggle with large groups, as the whole point is there isn't a defined process at all. Each team will inherently work differently, which introduces variance in expectations/output, which causes problems. Or does it? To me, absolutely not. Short iteration cycles means output is going to be incredibly varied anyway, as software is incredibly hard to create, and very rarely repeatable. But for those who believe variance is a problem, I would ask if they're looking for agile software development, or waterfall with biweekly demos. The actual answer is almost certainly the latter, but companies will never market themselves as such.
So how do you scale teams without a defined process? You define domains for teams to become the experts in. You work closely with teams to set expectations on release cycles. Expecting each team to be able to step into each domain and work at a similar pace means your expected pace needs to be incredibly slow. Three teams of ten who are allowed to work in the structure they want, on something they're experts in will 100% of the time outperform fifteen teams of ten working on mix and match feature sets across multiple domains/products.
Objectively, does it sound like a good idea to force your teammates into physical discomfort to make them talk less? Is the forced standup really worthwhile? Or, do we really believe you can't zone out while standing?
I work for a professional services company. We take clients across many industries, countries, backgrounds and mindsets. It is not their job to care about software development practices, necessarily. This means that a process created by team Hypnotoad may not be effective on team Toast (my team). This also means that processes implemented on one project on team Toast may not work for another. That needs to be okay. The goal is not to follow an agile process that's been purchased, but to build working software for our clients. My teammate and I found out the hard way that there is not one process to rule them all, even if it is team defined.
Josh and I worked for weeks on a project following the same process we'd defined for previous projects. Each morning there was a standup (that we never stand in), each sprint is two weeks, starting with a sprint planning/commitment ceremony and ending with a demo and retro. This process requires a client is aware of, and happy with a sprint commitment and the associated velocity, as well as desire to track that velocity long term. But what happens when a project doesn't fit that mold?
We threw it away. All of it. Josh and I removed literally every aspect of our development process. No more morning standup meetings, no more retrospectives, no sprint commitments, no velocity. Not even sprints. Josh and I show up to work every day (whether showing up is a metaphor for turning on a laptop or going into an office). Inevitably Josh and I communicate, as we're working very closely in the same domains on the same software. We talk to the client a lot, and each time we talk to him we get a better sense of his goals, his needs and his nice to haves. Each time we talk to we agree on some high priority stuff to do, and Josh and I privately divide that work up. No sprint planning, no client buy in, no velocity.
But That's Not Agile!!!!!11
It absolutely is. I would argue this process is infinitely more agile than any other form of development I've ever seen. Every few days we send our client a change log, but we don't even really have to as he uses our software every day. Our release cycle is whenever something is ready, it gets released.
The crazy part: the whole thing worked. Our client immediately became happier, as we stopped nagging him about Jira tickets and sprint commitments, of which he cared about not at all. Everything turned around when we threw it all away. We went from a struggling software team that wasn't making their client happy to a high performing, strong, well trusted team, all by reducing process, not adding it.
Throw it out. If every dev on a team isn't feeling totally satisfied with a process item or regular meeting, get rid of it. Less is more.
The one thing I'll add is that I think the frequent retrospective adds tremendous value, only if done correctly. I've been in lots of retros where the output for each item is "We'll try to be better about x", which means "This will continue to be painful for us, and we won't do anything about it." Retros can be a huge waste of time, but when they're not they're extremely important. Letting teammates discuss problems they're actually having, and letting a team decide how to solve them internally is extremely impactful. Teams that feel like they own their process work better.