the influential architect - carnegie mellon university...envato ‣ 4 groups with 3-8 software...

SATURN 2017

Title of the Presentation Goes Here © 2017 [Copyright Owner[s]]The Influential Architect © 2017 Sebastian von Conrad 1

The Influential ArchitectSucceeding At Scale Among Fully Autonomous Teams

Sebastian von Conrad

SATURN 2017

The Influential Architect © 2017 Sebastian von Conrad 2

👋

SATURN 2017


SATURN 2017


We’re a marketplacefor digital creatives.

SATURN 2017


Envato

SATURN 2017


Envato‣ Founded in 2006.

SATURN 2017



‣ Paid > $500m USD to our content authors.

SATURN 2017




‣ > 300 staff worldwide.

SATURN 2017





‣ > 100 people in the Technology team, reporting to CTO.

SATURN 2017





‣ > 100 people in the Technology team, reporting to CTO.

‣ 1 Software Architect. (Me!)

SATURN 2017


Envato Org ChartCTO

Senior Eng. Mgr.

Senior Eng. Mgr.

Senior Eng. Mgr.

Senior Eng. Mgr.

SoftwareArchitect

SATURN 2017


Envato

SATURN 2017


Envato‣ 4 Groups with 3-8 Software Engineering teams each.

SATURN 2017



‣ 23 Software Engineering teams in total:

SATURN 2017




‣ Every team is cross-functional (PO, UX, Frontend, Backend, DevOps.)

SATURN 2017





‣ Every team has 4-8 Engineers.

SATURN 2017






‣ Every team has a Engineering Team Lead.

SATURN 2017







‣ Every team has a Technical Lead (usually Team Lead or Senior Engineer.)

SATURN 2017








‣ One job: answering “why did you do it that way?”

SATURN 2017








‣ One job: answering “why did you do it that way?”

‣ 1 Software Architect. (Me!)

SATURN 2017


Envato

SATURN 2017


Envato‣ No central control and command over technology decisions.

SATURN 2017



‣ Each team is autonomous and make their own decisions.

SATURN 2017




‣ Including technologies (languages, databases, etc).

SATURN 2017





‣ Each team is accountable for their own decisions.

SATURN 2017






‣ Each team owns systems.

SATURN 2017







‣ Includes operational responsibility, budgets, etc.

SATURN 2017







‣ Includes operational responsibility, budgets, etc.

‣ Every system must be owned by a team.

SATURN 2017


My job is to improve the qualityof technical decisions.

Without any authority.

SATURN 2017


More often than not, it feels like my job is to fight entropy.

SATURN 2017


And I don’t scale very well.

SATURN 2017


😱

SATURN 2017


The only way I can affect meaningful change is to influence

the decision-makers.

SATURN 2017


I have to focus on people,not technology.

SATURN 2017


And get them to do my job for me.

Seriously.

SATURN 2017


SATURN 2017


We have anArchitecture Guild

SATURN 2017


Envato’s Architecture Guild

SATURN 2017


Envato’s Architecture Guild‣ It’s a group that discusses and determines things (we'll get to what later).

SATURN 2017



‣ It's a sounding board for technical decisions.

SATURN 2017




‣ It's a forum for share knowledge and learn from each other.

SATURN 2017





‣ Anyone in the Technology team is welcome to participate.

SATURN 2017





‣ Anyone in the Technology team is welcome to participate.

‣ Some participants are mandatory. (I get to decide who.)

SATURN 2017


Let’s talk about the things that we do

SATURN 2017


Agenda

SATURN 2017


Agenda‣ Transparent Decision-making

SATURN 2017



‣ The Light on the Hill

SATURN 2017




‣ Architectural Principles

SATURN 2017





‣ Sensible Defaults

SATURN 2017





‣ Sensible Defaults

‣ Strategic Design

SATURN 2017


Disclaimer: the plural of anecdote is not data.

SATURN 2017


Disclaimer: I don’t claim to have come up with

anything new.

SATURN 2017


TransparentDecision-making

SATURN 2017


Document and Communicate Intentions.

SATURN 2017


Project docs.

SATURN 2017


Project Docs

SATURN 2017


Project Docs‣ 2-6 pages.

SATURN 2017



‣ Table of Contents:

SATURN 2017




‣ What’s this about?

SATURN 2017





‣ Key constraints

SATURN 2017





‣ Key constraints

‣ Quality Attributes we’re optimising for

SATURN 2017





‣ Key constraints


‣ Quality Attributes we’re not optimising for

SATURN 2017





‣ Key constraints



‣ Solution summary

SATURN 2017





‣ Key constraints



‣ Solution summary

‣ Effects on existing systems

SATURN 2017


Project Docs

SATURN 2017


Project Docs‣ Doesn’t take long to create—a few hours.

SATURN 2017



‣ Timing is everything.

SATURN 2017




‣ “Ghost writing” services offered.

SATURN 2017




‣ “Ghost writing” services offered.

‣ Interview team, write for them.

SATURN 2017


Architect as Bottleneck

SATURN 2017


Architect as Bottleneck‣ “Ghost writing” doesn’t scale.

SATURN 2017



‣ Team Lead: “I feel like you’re more ‘ghost’ than ‘writer’.”

SATURN 2017




‣ Doesn’t have to scale.

SATURN 2017





‣ Establish precedent.

SATURN 2017






‣ Create templates.

SATURN 2017







‣ Share knowledge

SATURN 2017







‣ Share knowledge

‣ Make teams self-sufficient.

SATURN 2017


Project Docs Feedback

SATURN 2017


Project Docs Feedback‣ Anyone can comment.

SATURN 2017



‣ Everyone comments.

SATURN 2017




‣ Buy-in from technical stakeholders is desired.

SATURN 2017





‣ Revisions are expected.

SATURN 2017






‣ Quick iteration.

SATURN 2017







‣ If you miss your chance to comment, tough.

SATURN 2017







‣ If you miss your chance to comment, tough.

‣ Tool: Google Docs.

SATURN 2017


Side note: we’re also experimenting with ADRs, go watch Michael Keeling’s talk.

SATURN 2017


The Light on The Hill

SATURN 2017


Teams need a Technical Vision.

SATURN 2017


…it’s not a roadmap!

SATURN 2017


Light on the Hill

SATURN 2017


Light on the Hill‣ A vision, not a roadmap.

SATURN 2017



‣ Our best guess of what the future looks like.

SATURN 2017




‣ Guides technical decision-making.

SATURN 2017





‣ Keeps teams honest: are we heading in the right direction?

SATURN 2017






‣ If not (which is okay), at least we’re conscious of it and do it for the right reasons.

SATURN 2017







‣ Each group has their own Light on the Hill.

SATURN 2017







‣ Each group has their own Light on the Hill.

‣ The group’s Senior Engineering Manager (or their designated delegates) owns the Light on the Hill.

SATURN 2017


Great results, through business and product stakeholder buy-in.

SATURN 2017


Expressing a Light on the Hill

SATURN 2017


Expressing a Light on the Hill‣ Initially, a big giant doc/diagram for each group.

SATURN 2017



‣ Yeah, that didn’t work.

SATURN 2017




‣ Hard to maintain: got out of date quickly.

SATURN 2017





‣ Information overload.

SATURN 2017





‣ Information overload.

‣ Difficult for technical folks, much less non-technical stakeholders, to wrap their heads around it.

SATURN 2017


Instead, we sliced by capability.

SATURN 2017


…and used visual storytelling.

SATURN 2017


The Anatomy of a Story

SATURN 2017


The Anatomy of a Story‣ In the past…

SATURN 2017



‣ Then something happened…

SATURN 2017




‣ Now we’re…

SATURN 2017




‣ Now we’re…

‣ So that in the future…

SATURN 2017


Let’s use a real example!

Federated User InformationLight on the Hill

MarketIn the beginning, products like Market had all information about their users in their own databases.

MarketThen when we were building Studio, we decided that we would benefit from sharing user information between products.

We built account.envato.com (SSO) to provide a unified sign up/in experience and share user information between products.

Studio

SSO

MarketOf course, not all information made its way over to SSO. Both Market and Studio also collected their own information about users.

Studio

SSO

MarketOver time, more of the products we built were integrated with SSO, all the while also capturing their own information relevant to them.

Studio

SSO

Elements ...

MarketWe also wanted to share more information than just basic user details, such as billing details and tax information.

We built a system called Identity to hold this information.

Now, to get the information about users product wanted, they had to integrate with both Identity and SSO.

Studio

SSO

Elements ...

Identity

MarketWe are now finding that there is information in the products themselves that other products are interested in.

For example, Market might want to know if a user is an Elements subscriber so they can give discounts on Market purchases.

Studio

SSO

Elements ...

Identity

What we’ve ended up with is that different products and systems each hold a different aspect of what we know about a user.

Some of the systems that have key information are ones we haven’t built ourselves, like Zendesk or Discourse (our community forums).

SSO Identity Elements

Market Tuts+ Studio

Zendesk Salesforce Discourse

With information about users scattered around different products and systems, giving products and systems the information they need is quickly becoming a problem.

Lots of integration points and interdependencies makes systems harder to manage and change.

X

In the future, we will build a new system (tentatively called X) which holds a federated view of users.

X

To get information into X, a Watcher will “watch” a particular source of user information.

The Watcher will send the information it retrieves from the source to X.

X will only ever have a copy of the information. The source will still be the source of truth.

Watcher

Source

Pull

Push

There will be many Watchers that each watch different systems for information.

Some Watchers may watch several systems, while others only watch one.SSOIdentity

Watcher WatcherWatcher

Elements

Market

Watcher Watcher

Elements

Zendesk

X

The many Watchers will push the information they retrieve from the source systems to X.

Watcher WatcherWatcher Watcher

In collecting this data, X can present a federated view of the information we have about the user.

Over time, products will start asking X for information about users, rather than integrating with every other system directly.

Studio ...

X

Market Elements

Identity SSO

X

And in the end, the Light on the Hill is for all products and systems to only ever ask X for information about users.

Market Elements Studio ...

SATURN 2017


The intended audience is not technical folks.

SATURN 2017


It’s to get broad agreement on technical vision with business and product owners.

SATURN 2017


This meme has caught on.

SATURN 2017


Developing a Light on the Hill

SATURN 2017


Developing a Light on the Hill‣ Workshop with selected participants:

SATURN 2017



‣ Key tech and product people from group.

SATURN 2017




‣ Stakeholders from other dependent groups.

SATURN 2017





‣ Senior stakeholders (e.g. CTO, CFO) if interested.

SATURN 2017






‣ Outcome is a draft.

SATURN 2017







‣ Share with Architecture Guild for feedback, similar to Project Docs.

SATURN 2017








‣ Once ratified, share far and wide. (Demos, exec team meetings, etc.)

SATURN 2017








‣ Once ratified, share far and wide. (Demos, exec team meetings, etc.)

‣ Refer to them often.

SATURN 2017


Architectural Principles

SATURN 2017


Broad, high-level principles we want all teams to follow.

SATURN 2017



SATURN 2017


Architectural Principles‣ 7 ± 2.

SATURN 2017



‣ Expressed as value statements.

SATURN 2017




‣ Technology/pattern agnostic.

SATURN 2017





‣ Examples:

SATURN 2017





‣ Examples:

‣ We prefer asynchronous communication.

SATURN 2017





‣ Examples:


‣ We keep dependencies acyclic and stable.

SATURN 2017





‣ Examples:


‣ We keep dependencies acyclic and stable.

‣ We define Quality Attributes for systems and design with them in mind.

SATURN 2017



SATURN 2017


Architectural Principles‣ They apply to everything technical decision we make.

SATURN 2017



‣ They are vehicles for accountability.

SATURN 2017



‣ They are vehicles for accountability.

‣ Anyone is able to hold anyone else accountable to them.

SATURN 2017


Architectural Principles Process

SATURN 2017


Architectural Principles Process‣ Anonymous suggestion box.

SATURN 2017



‣ Curation => first draft.

SATURN 2017




‣ Deep discussion on each one => Second draft => Third draft.

SATURN 2017





‣ Buy-in was crucial.

SATURN 2017





‣ Buy-in was crucial.

‣ Once finalised, published.

SATURN 2017


Sensible Defaults

SATURN 2017


We see patterns emerge across teams.

SATURN 2017


Same problem, different team.

SATURN 2017


Don’t want to reinvent the wheel.

SATURN 2017


Sensible Defaults

SATURN 2017


Sensible Defaults‣ Off-the-shelf solutions to common problems.

SATURN 2017



‣ E.g. API Design, Authentication, Logging, 3rd Party Integration, Identifying and Handling Sensitive Data.

SATURN 2017




‣ Typically originates in practice, not theory.

SATURN 2017





‣ Described as specifications.

SATURN 2017






‣ Specifications frequently lead to libraries.

SATURN 2017







‣ Includes why as well as what, or we may miss an opportunity to teach.

SATURN 2017







‣ Includes why as well as what, or we may miss an opportunity to teach.

‣ The why also enables the sensible default to be challenged.

SATURN 2017


Sensible Default example: Authentication‣ MUST use request signing.

‣ MUST sign either the entire request body or a hash of the entire request body.

‣ Hashing is RECOMMENDED when the request body is large. Use SHA256 or better for hashing.

‣ MUST sign the following request headers (if they are sent): Content-Length, Content-Type, Date, User-Agent

‣ MUST sign the request path.

‣ MUST set an expiry time for the request.

‣ SHOULD sign any other request headers you deem appropriate.

‣ MAY send a unique identifier for the request and validate server-side to prevent replay attacks.

‣ MUST use TLS/SSL for the request.

‣ SHOULD obtain an externally verifiable certificate for TLS/SSL.

‣ SHOULD whitelist IP addresses of expected clients and fail authentication for requests from other IPs.

‣ MUST drop any requests that failed authentication.

‣ MUST log any requests that failed authentication.

‣ MAY raise on-call alerts for requests that failed authentication if warranted.

SATURN 2017


Sensible Defaults

SATURN 2017


Sensible Defaults‣ Sensible Defaults are recommended, but never mandated.

SATURN 2017



‣ You’ll never get in trouble for using one.

SATURN 2017




‣ If you choose not to use one, you’ll be asked why.

SATURN 2017





‣ We don’t want to stymie innovation.

SATURN 2017






‣ Have a better idea? Go for it!

SATURN 2017






‣ Have a better idea? Go for it!

‣ We can update the default.

SATURN 2017


Sensible Defaults Process

SATURN 2017


Sensible Defaults Process‣ Anyone can write a Sensible Default.

SATURN 2017


Sensible Defaults Process‣ Anyone can write a Sensible Default.

‣ Same process as for everything else: open discussion and iteration.

SATURN 2017


Strategic Design

SATURN 2017


This may be controversial…

SATURN 2017


😬

SATURN 2017


Architecture exists to serve the needs of the business.

SATURN 2017


A better architecture allows us to go faster and say “yes” more often.

SATURN 2017


As such, our architectural goals must follow your business goals.

SATURN 2017


CEO says “I want to do X…”

SATURN 2017


Here’s the thing: sometimes the shortcut is right.

SATURN 2017


I often find myself representing the idealism, out of necessity.

SATURN 2017


The number one trait I look for in Tech Leads is balance between

Idealism and Pragmatism.

SATURN 2017


Strategic Design

SATURN 2017


Strategic Design‣ Understand business objectives and strategy.

SATURN 2017



‣ Weigh tradeoffs.

SATURN 2017




‣ The design has to be proportionate to the business value.

SATURN 2017




‣ The design has to be proportionate to the business value.

‣ Use the Strategic Design graph.

SATURN 2017


Strategic Design

Competitive advantage

Com

plex

ity

Buy

WTF?KISS

SATURN 2017


In Summary

SATURN 2017


I’m not gonna lie…

SATURN 2017


…sometimes my job would be easier if I could just tell people what to do.

SATURN 2017


…or at least act as a gatekeeper for their decisions.

SATURN 2017


But with 23 autonomous teams,that doesn’t scale.

SATURN 2017


So how do I improve the quality of our Technical Decision-making at scale?

SATURN 2017


What controls do I put in place?

SATURN 2017


By making teams accountable to each other.

SATURN 2017


Sometimes the conversationsare confronting.

SATURN 2017


“The Architecture Guild is where you go when your impostor syndrome has been

dormant for a while.”- Envato Tech Lead

SATURN 2017


“The Architecture Guild is like baringyour soul to the world.”

- Envato Senior Engineer

SATURN 2017


But if I’ve done my job right, the difference is not in the decisions

themselves…

SATURN 2017


…but the people who make them.

SATURN 2017


They will express their technical vision as Lights on the Hill.

SATURN 2017


They will communicate their intent via Project Docs.

SATURN 2017


They will adhere to Architectural Principles.

SATURN 2017


They will use (or don't use)Sensible Defaults.

SATURN 2017


They will be Strategic in their Designs.

SATURN 2017


Thank you.

the influential architect - carnegie mellon university...envato ‣ 4 groups with 3-8 software...

Documents