http://netponto.org #netponto

50ª Reunião Lisboa – 22.11.2014

Architecting a Large Software Project – Lessons LearnedJoão Pedro Martins

João Pedro “jota” Martins

CTO @ |create|it|

Software Architect

TechEd 2006 – “Iron Architect” Winner

BizTalk Server MVP 2006-2011

Co-founder - GASP + APPU

Windows Azure Insider

|create|it|

Founded 2001 @ Lisboa

(Gourmet) Systems Integrator

Team of (proud) 26

Azure - BizTalk - Office 365 – SharePoint

Umbraco - NopCommerce

What’s this session about?

Architecting a

Large

Software Project –

Lessons Learned

Overview

Banking application for internal use (>3000p)

Web-based, running inside a Windows Forms shell

Started Jun-2011, R1 installed Jul-2014

Scrum - 50 sprints

12 man-years in the making

Integrates with 11 systems

Team of 6 + customer team ~8 + other teams ~10

THE PROJECT

Statistics (Mar-2014, Sprint 41)

Application code - LOC

32.525 C#

16.782 Javascript

30.664 Rule Engine XML

8.445 T-SQL

9.552 HTML (MVC3)

Automated Tests - LOC

GeneratedCode - DAL And also…

40.243 C#

26.878 T-SQL Inserts (test data)

1.299 tests

80% codecoverage

146.009 C#

53.427 T-SQL Stored Procedures

60 Visual Studio Projects

101 SQL Tables

17 WCF Services with 190 Operations

59 MVC Views/Partial Views

100% StyleCop compliance

THE PROJECT

finally { … }

~500 user stories

723 issues: 90% closed, 65% bugs

Conceptual Architectureand Technologies

THE PROJECT

browser

Controller

View

ViewModel

dadosserviçosweb

Controller

View

Model

business logic

rule engine

integration

tables + views

data access

stored procedures

browser web services data

NET 4.0

CODESMITH +

NETTIERS TEMPLATES

T-SQL

SQL SERVER + VS

DATA TOOLS

SSMS

NET 4.0

WCF

NXBRE

LOG4NET

UNITY (ENTLIB)

VS UNIT TESTING

RHINO.MOCKS

SPREADSHEET.NET

QUICKGRAPH.NET

TOOLKIT CREATE

.NET 4.0

ASP.NET MVC3

LOG4NET

JSON.NET

UNITY (ENTLIB)

TOOLKIT CREATE

BALSAMIQ MOCKUPS

HTML + CSS

JQUERY

KNOCKOUTJS

UNDERSCOREJS

KENDOUI

MOMENTJS

TOASTR

2,5 years ago…FRIENDLYREMINDER

The “Secrets” then…

Scrum

Agility

Frequent releases

Notion of progress

Burndown

User Interface

Motivated Team Focus

Pretty

Simple

Innovative

Team principles - quality

Continuous improvement

Individual initiative

Redundancy

Individual strong points

Work-life balance

Challenges

Recent technology

Workspace

Relationships

Enough context!

Now, the lessons…

Classes with too many responsibilities

LEARNING #1

Service Implementation

Business Manager

Data Access Layer

IOperationInterface

Data Contracts

Business Manager

These

go

tto

o larg

e

WCF Physical Hosting (.svc’s)

How to work around this?

SOLID principles - Single Responsibility

Domain Driven Design – vertical slices of behaviour

Separation of Concerns – partial classes

When in doubt:

• Create another class

• Use interfaces

LEARNING#1

Logging

Do you have detailed logging enabled in your

production environments?

LEARNING#2

Is this useful?

Logging

Instead of this…

… try this…

LEARNING #2

Logging -hints

In some cases, you can’t log details of the operation.

Use correlation - relate log entries in the same context.

Beware of impact on performance.

Beware of storage space required. DO cleanup.*not a problem on the cloud

To read text logs, try this tool: LogExpert

LEARNING #2

Interfaces vs Inheritance

“Why extends is evil - Improve your code by replacing

concrete base classes with interfaces” (Allen Holub)http://www.javaworld.com/article/2073649/core-java/why-extends-is-evil.html

LEARNING #3

Dependency injection interfaces.

Very limited use of OO inheritance- Base Data Contracts with minimum property set (e.g., id + name)

- Less than a handful of uses in total

The bad part? F12 navigates to the interface, not the

implementation

Dependency Injection

Indirection when instantiating objects:

- Container builds and reuses objects

- Supports mocking for automated tests

Interception:

- Cache

- Logging

- Profiling

- Exception Shielding

Fully configurable in .config

LEARNING #4

Caller

Target

Interceptor – Call handler

Dependency Injection pitfalls

Initial setup can be demanding (skills+time)

Programming configurations (complex debugging)

Impact on runtime performance

LEARNING #4

(cont)

Call handler configuration

Cache

Cache transparently via interception +

configuration.

Cache before accessing the network (CPU is

cheap).

Designed for 3 freshness configs.

LEARNING #5

Presentation Layer

Cache: ooops!

Business information presented must be accurate –

and data is not stable in external systems.

Very limited set of external reference data.

User authorization must be considered.

Transparent, configuration-based cache, is convenient -

however, you can’t selectively expire contents!

LEARNING #5

Automated tests

Requirement: 80% coverage by automated tests in

service layer.

Team principle: the AGILE team is not afraid to change

any piece of code for fear of breaking something.

Approach: service-level, end-to-end tests

- Visual Studio Tests

- Not unit tests

- Depend on external data

- Sprint backlog: One service operation, one test set

LEARNING #6

Automated tests – mmmm…

Test suite takes too long to run (~2h) :

- SQL scripts SQL Database Snapshots

- Service layer tests Business layer tests

External data not stable – mocks

But: how to test complex business cases dependent on

external data of which we can’t be sure?

LEARNING #6

Automated tests – more mmmm…

VS2012 test runner worse than VS2010 (!)

Have a Test King in the team

Smart asserts improve code-coverage:

Tool recommendation: SSMS to generate T-SQL from data

LEARNING #6

Code Conventions

Agree on coding conventions and stylecop compliance

at start of project.

Architect/Tech Lead name all the main artifacts: service

contracts, database artifacts, etc. Strive for consistency.

LEARNING #7

Code Conventions notes

Focus on code legibility:

- Don’t use var for non-anonymous types

- Don’t overdo Linq statements

- Comment your code

Mistakes will happen, and renames will be needed

(mixing PT with EN is frequent).

Uniformize verbs in services/methods.

Do NOT argue tab size. In doubt, use defaults.

LEARNING #7

Negotiation… with your team, and with the customer

Always voice your opinion, focusing on what’s best for

the project. Create a trust relationship.

When your approach is not followed, and you are sure

you are right, present objective arguments – don’t be

emotional. Argue for as long as you must, but not

longer.

Accept defeat, make compromises.

LEARNING #8

Negotiation… some more notes

Be attentive of the other’s possible hidden motivations,

but be careful in exposing them.

Consensus is not always possible. Your options will be

questioned, and sometimes you will be wrong.

Remember the 3 views of architecture: theoretical,

pragmatic, implementable

Ask open questions.

LEARNING #8

Functional team/domain experts

They are your peers, and part of the team. You

depend on well written and clear user stories.

Domain experts that understand Scrum and

team/time constraints make the difference.

Rely heavily on them and their tools. When you

don’t understand, ask questions until you do.

LEARNING #9

Functional Team do’s and don’ts

Sometimes the way a story is written crystalizes a way

of implementation.

Some stories will be hard to understand and

decompose into tasks. Ask for clarifications and don’t

implement blindly.

Business context is sometimes missing.

Tendency to “follow the old ways”.

Tendency to abide to single-user/hierarchical requests.

Use Your Brain: design elegantly

You are not paid to write code, you are paid to think

and communicate.

Think things through before committing to a solution.

Try to isolate and design autonomous and change-

tolerant components.

Step back, look at the larger picture. As an Architect,

you DO NOT have to be a technical expert in

everything: focus on capabilities and structure.

LEARNING #10

Impediments to using your brain

Interruptions, no whiteboard, too much

noise, lack of natural light, music on

headphones, time or budget pressure, too

much coffee, personal problems, lack of

sleep, …

What’s your style: collaborate then design, or design

and then collaborate? Isolate yourself to design. Make

drawings, and the document your proposed solution.

* Thinking is hard work.

Revisiting 4 technical choices

KnockoutJS or MVC3?

Took time to decide and spike,

there was an initial setback with

KO and adoption was reversed.

2nd attempt and investment

proved correct.

NxBRE Rules Engine

Quickgraph.net Distributed Cache

XML-based rules engine DLL.

XML file can be replaced without

recompilation. Works fine and is

fast, but hard to code and read.

Jury is still out.

Formal Architecture feedback

against use was tacitly dismissed as

non-pragmatic/expensive, and

package was used.

VelocityCTP3 was refused as non-

supported. AppFabric not available

in Windows version. Oracle

Coherence never provided.

Lightning round

Your team is an

extension of your

body.

3 (physical) layers & no

distributed cache mean no

real-time features

(e.g., SignalR)

Know thy user’s pc:

@start, 1GB RAM, IE8, WinXp, 1024px

Javascript, IE8, 1GB RAM recipe for

disaster.

Use extension methods – don’t

pollute your classes with

internal – auxiliary - methods

* and kill those «helper» classes, too

Create and

discuss

mockups!

Tool recommenda-

tion: Balsamiq

Mockups

Be lazy. Don’t waste precious

time coding your own, special,

data-access layer/....* also, scavenge codeplex+github+npm+… for assets to buy or

reuse

Use an issue tracker, designate someone do to

the triage, and configure mail alerts, your

pages/modules, team, and sprints.

Teach the customer

how to use it for bugs

& enhancements.

Visual Studio Database

Projects and Schema Compare

are priceless features. Use

them.* start today

Use diagrams to communicate

and structure your thoughts.

Usability tests are simple! Just

looking at your users’ work

uncovers problems and ideas

for improvement.

Just 3 more before I go…

Humans make mistakes. Scripts don’t. Use scripts and

obsessively automate repetitive tasks or installations.

Know your branches protocol, shelves, labels,

versioning (just use Microsoft’s recommendation).

Innovating and surprising your customer, and the

cherry on the cake, makes a world of difference.

Closing message

It’s an architect job to address the

customers’ needs, deliver quality

(build cathedrals), and learn

constantly.

Hope I helped.

ideas, questions, experiences?

Thanks for having me!

(+351) 96 782 5537jota@create.ptblogit.create.pt/joaomartins/twitter.com/lokijotapt.linkedin.com/in/joaopedromartins/

Patrocinadores “GOLD”

Twitter: @PTMicrosoft

http://www.microsoft.com/portugal

Twitter: @FindMoreC

http://www.findmore.eu

Patrocinadores “Silver”

Patrocinadores “Bronze”

http://bit.ly/netponto-aval-50

* Para quem não puder preencher durante a reunião, iremos enviar um email com o link à tarde

Próximas reuniões presenciais

22/11/2014 –Novembro –50ª Reunião! (Lisboa)

13/12/2014 –Dezembro (Lisboa)

24/01/2015 – Janeiro (Lisboa)

Reserva estes dias na agenda! :)