IHateSpaghetti {code}

Insights to development, deployment and management in large scale application

OK, as promised in my previous post I'm going to write above some of the insights I had while working in large scale project (man power, equipment you name it).

I'll start from development

In this project a lot of the code is automatically generated (about 70%-90%).

As such there is a special team that creates these templates by development teams needs.  I think it's a good idea because creating templates requires special type of developers and when you have a dedicated team for this job, the developers in that team are constantly improving their skills it this area of expertise and create high quality templates in reasonable time.

On the other hand when most of your code is generated for you and you have to implement or inject a function or two it leads to poor code ownership and poor understanding of the overall picture. When something breaks (and it does) the developers of the higher levels of the application layers will have more difficulties identifying the problem and nearly no chance of overcome it.

Developing large scale application usually take time, and as times goes by the mobility of the working force take key factor in knowledge-loss scenarios. It's hard to overcome this issue so try to write more maintainable code – if most of you code/logic is parsed/consumed and not compiled it will be very hard to maintain it especially when developers leave, so think if using a lot of configuration and dynamically loading it during run time is worth the maintainability penalty – in most of the cases you can generate code from that configuration in a pre build event (or some way with equivalent effect) that  way you'll be able to build and run tests against it. This level of compliable-configuration will allow your code to be more understandable (especially if you'll put some remark in the header of the generated code stating that it was auto generated).

Key points:

·         If you generate code, think about who needs to implement it, how long will the project last (for long periods your can risk knowledge loss when people leave).

·         Consider using more OO instead of code generation, and when generating prefer generating partial classes instead of regular classes

Now regarding the builds and deployment

I think that in early stages of the project you tend to think more about the deployment than about builds (if at all), I disagree with this. You don’t have to be an agile "fan" in order to adopt CI (continuous integration). You can achieve high flexibility in testing the product with all the latest bug fixes and features but only if you have CI (by dropping specific DLL's and testing them you can easily have .NET-type DLL hell during integrated tests!).

As the project advances you have more projects/solutions to build, test and pack all this takes a lot of very precious time. In more than one occasion we had forced-unemployment because developer(s) was/were waiting for a completion of a specific build.

Merges between a lot of developers (and teams) can be very problematic (builds might break, code might be overwritten or event forgotten to be merged – you need to plan and form a methodology based on the hierarchy of the teams their development responsibilities and architectural design of the application.

·         Builds can take most of the time in the later stages. Plan towards fast incremental builds (by detecting that no changes been made for specific assembly and not compiling it).

And now a little bit about Management

In large scale application you have (usually) a lot of developers that writing a lot (hopefully good) code. You need to constantly CR (code reviews), developers tend to write "temp" code during high-pressure periods and that code tends to be "const" as time passes and you even start joking about the horrible code and this will eventually lead you to not necessarily good decisions from software entering perspective and so on and so on, refactor wisely and you'll benefit from it on the long run.

 When you have a lot of teams/layers/developers you need some kind of a way to integrate all the features/changes/bug fixes together, integration teams now becoming more popular but I think it can be even better. In my opinion integration team are "all knowing read-only" developers ("all knowing" - from application modules and architecture perspective, and "read only" because they don’t modify the code and most of the time don't develop). They should be aware of all the major features and special types of configurations in all the modules, perhaps by integrating them as part of architecture team or infrastructure team, this way they will gain the application-specific knowledge more easily. In either way this team should be highly involved with the application deployment and configuration.

·         Perform CR, refactor when needed before it becomes de facto!

Next I'll be writing about all kinds of interesting bits from my experience working in that project.