Teams in my company face the same types of problems. We are building projects with a large number of moving parts and architectural layers that make it difficult to create a working product early on. Additionally, there are often specialty resources that need to be scheduled or slightly out of synch with the team. Some approaches we ve taken are below It has been challenging, but these approaches seem to be helping.

Build as vertically as possible

• In other words, strive to have something working, end to end as quickly as possible. We typically get there a few sprints in on a 9-16 month project.
• You ll often find a significant number of layers can be mocked or held back.
• Often, the initial customer facing components are place holders. We create a limited bit of functionality that is something like what the customer wants, but is likely to be very different in the final project. This allows us to prove the rest of the product at a system level and provide visibility from a system perspective.

Separate base architecture from the product

Our early sprints are often centered around infrastructure/architecture. For example, threading subsystems, performance monitoring, communications and test frameworks.

• Treat the subsystems as separate deliverables
• Fully define each subsystem
• Complete (truly complete, not just a partial implementation) each subsystem
• Load test each subsystem within the context of how it will be used in the final product

Make your first iteration to be dedicated to architectural design, including the identification of the necessary components and the definition of the relationships and communications between them.

Once you have a clear picture of how the components interact, build the skeleton of each one. That is, implement "stub" components that just have the communication part on place, and the rest of the functionnality just do nothing or return test data. Have an interation dedicated to this task (including testing the component communication mechanisms) as well.

Then you can plan iterations to fully develop each component in the appropriate order, so that the system can grow in a ordered way.

TDD - iterate with incomplete parts after writing tests. Mock the bits that aren t ready. Sounds exciting.

It is unlikely that each layer needs to be complete for it to be usable by the other layers - for example the persistence layer could just serialize objects to a file initially, and converted to use a database when the need arises. I would look at implementing the minimum of each layer needed by the initial stories and fleshed out to add functionality as the the system grows.

Growing a system this way means you only implement the functionality you need, rather than all the functionality you think you may need at some indeterminate time in the future.

If you cannot break the large project into smaller parts that are useful (i.e. enable some use cases) on their own, agile probably won t help you that much in this project. You can pick some techniques like pair programming, refactoring etc. but the overall planning has do be done in a conventional way.