composing project archetyps with sbt autoplugins

Composing Project Archetypes with SBT

AutoPluginsMark Schaake

allenai.orgcontribute to humanity through high-impact AI research and engineering

Composing Project Archetypes with SBT

AutoPluginsMark Schaake

allenai.orgcontribute to humanity through high-impact AI research and engineering

Combatting Multiple Build Maintenance Hell with Archetype AutoPlugins

Mark Schaake

allenai.orgcontribute to humanity through high-impact AI research and engineering

When you have very limited visibility and control over many project builds to the point

where you feel paralyzed to maintaining cross-project consistency.

(Multiple Build Maintenance Hell)MBMH

visibility: what libraries, plugins, versions? control: how to enforce team standards, upgrades? consistency: minimize context switching

SOA, Inc.

SOA architecture

Scala, Akka, Spray

Svc A

Plugins Projects Dependencies

Svc A

Plugins Projects Dependencies

packager 0.6.0

scalariform 1.3

// native packager for generating deployable artifacts addSbtPlugin( "com.typesafe.sbt" % "sbt-native-packager" % "0.6.0") // format our code in SBT instead of our editors addSbtPlugin( "com.typesafe.sbt" % "sbt-scalariform" % "1.3.0")

project/plugins.sbt

Svc A

Plugins Projects Dependencies

packager 0.6.0

scalariform 1.3

// native packager for generating deployable artifacts addSbtPlugin( "com.typesafe.sbt" % "sbt-native-packager" % "0.6.0") // format our code in SBT instead of our editors addSbtPlugin( "com.typesafe.sbt" % "sbt-scalariform" % "1.3.0")

project/plugins.sbt

akka 2.3.2

spray 1.3.0

dispatch 0.11.1

scala 2.10.2

organization := "com.soa" name := "service-a" scalaVersion := "2.10.2" libraryDependencies ++= Seq( "io.spray" % "spray-can" % "1.3.0", "io.spray" % "spray-routing" % "1.3.0", "com.typesafe.akka" %% "akka-actor" % "2.3.2", "com.typesafe.akka" %% "akka-slf4j" % "2.3.2", "net.databinder" %% "dispatch-core" % "0.11.1", "ch.qos.logback" % "logback-classic" % "1.1.2" ... )

build.sbt

// native packager for generating deployable artifacts addSbtPlugin( "com.typesafe.sbt" % "sbt-native-packager" % "0.6.0") // format our code in SBT instead of our editors addSbtPlugin( "com.typesafe.sbt" % "sbt-scalariform" % "1.3.0")

// native packager for generating deployable artifacts addSbtPlugin( "com.typesafe.sbt" % "sbt-native-packager" % "0.6.0") // format our code in SBT instead of our editors addSbtPlugin( "com.typesafe.sbt" % "sbt-scalariform" % 1.3.0")

addSbtPlugin( org.scalastyle" %% "scalastyle" % 0.5.0")

Svc A

Plugins Projects Dependencies

akka 2.3.2

spray 1.3.0

dispatch 0.11.1Svc B

akka 2.3.4

spray 1.3.1

scala 2.10.2

scala 2.11.1

packager 0.6.0

scalastyle

scalariform 1.3

organization := "com.soa" name := "service-a" scalaVersion := "2.10.2" libraryDependencies ++= Seq( "io.spray" % "spray-can" % "1.3.0", "io.spray" % "spray-routing" % "1.3.0", "com.typesafe.akka" %% "akka-actor" % "2.3.2", "com.typesafe.akka" %% "akka-slf4j" % "2.3.2", "net.databinder" %% "dispatch-core" % "0.11.1", "ch.qos.logback" % "logback-classic" % "1.1.2" ... )

build.sbt

organization := "com.soa" name := "service-b" scalaVersion := "2.11.1" libraryDependencies ++= Seq( "io.spray" % "spray-can" % "1.3.1", "io.spray" % "spray-routing" % "1.3.1", "com.typesafe.akka" %% "akka-actor" % "2.3.4", "com.typesafe.akka" %% "akka-slf4j" % "2.3.4", "net.databinder" %% "dispatch-core" % "0.11.1", "ch.qos.logback" % "logback-classic" % "1.1.2" ... )

project/plugins.sbt

Svc A

Plugins Projects Dependencies

akka 2.3.2

spray 1.3.0

dispatch 0.11.1Svc B

akka 2.3.4

spray-client 1.3.2

spray 1.3.1Svc C

scala 2.10.2

scala 2.11.1

akka 2.3.6

spray 1.3.2

packager 0.6.0

scalastyle

packager 0.7.2

scalariform 1.3

scalariform 1.5

Svc A

Plugins Projects Dependencies

akka 2.3.2

spray 1.3.0

dispatch 0.11.1Svc B

akka 2.3.4

spray-client 1.3.2

spray 1.3.1Svc C

scala 2.10.2

scala 2.11.1

akka 2.3.6

spray 1.3.2

Svc D

packager 0.6.0

scalastyle

packager 0.7.2

scalariform 1.3

scalariform 1.5

Svc A

Plugins Projects Dependencies

akka 2.3.2

spray 1.3.0

dispatch 0.11.1Svc B

akka 2.3.4

spray-client 1.3.2

spray 1.3.1Svc C

scala 2.10.2

scala 2.11.1

akka 2.3.6

spray 1.3.2

Svc D

packager 0.6.0

scalastyle

packager 0.7.2

scalariform 1.3

scalariform 1.5 Lib X

Lib Y

CLI Z scopt

sbt-release

assembly

project/plugins.sbt 6

build.sbt 50

Project build LOC 56

x 26 projects

= 1,456 LOC!

Avg Project Build LOC

and mostly boilerplate

project/plugins.sbt 6

build.sbt 50

Project build LOC 56

x 26 projects

= 1,456 LOC!

Avg Project Build LOC

and mostly boilerplateMBMH!

and MBMH

Build

LOC

500

1,000

1,500

2,000

2,500

Proje

cts

15

30

45

60

Jan March May July Sept Nov

Projects Build LOC

and MBMH

Build

LOC

500

1,000

1,500

2,000

2,500

Proje

cts

15

30

45

60

Jan March May July Sept Nov

Projects Build LOC

M

Build

LOC

500

1,000

1,500

2,000

2,500

Proje

cts

15

30

45

60

Jan March May July Sept Nov

Projects Build LOC

MBMH

and MBMH

Yay!

Solving MBMH

Solving Maximize consistency across projects

MBMH

Solving Maximize consistency across projects

Minimize build complexity (LOC)

MBMH

Solving Maximize consistency across projects

Minimize build complexity (LOC)

Maximize agility to evolve standards

MBMH

Solving Maximize consistency across projects

Minimize build complexity (LOC)

Maximize agility to evolve standards

Allow for stragglers (dont force upgrades)

MBMH

Solving Maximize consistency across projects

Minimize build complexity (LOC)

Maximize agility to evolve standards

Allow for stragglers (dont force upgrades)

Easy build upgrade path

MBMH

Other Considerations

Other Considerations

Archetype settings shared by similar projects

Other Considerations

Archetype settings shared by similar projects

Core settings common to all projects

Other Considerations

Archetype settings shared by similar projects

Core settings common to all projects

Formatting and Style

Other Considerations

Archetype settings shared by similar projects

Core settings common to all projects

Formatting and Style

Generate Git version resource

Solution: Archetype SBT Plugins

Solution: Archetype SBT Plugins

Projects enable a single (versioned) archetype plugin

Solution: Archetype SBT Plugins

Projects enable a single (versioned) archetype plugin

Archetype plugin provides:

core build settings (style, scala version, etc.)

Solution: Archetype SBT Plugins

Projects enable a single (versioned) archetype plugin

Archetype plugin provides:

core build settings (style, scala version, etc.)

archetype build settings (deploy, publish, etc.)

Solution: Archetype SBT Plugins

Projects enable a single (versioned) archetype plugin

Archetype plugin provides:

core build settings (style, scala version, etc.)

archetype build settings (deploy, publish, etc.)

archetype dependencies (e.g. spray, akka, scopt)

Solution: Archetype SBT Plugins

Projects enable a single (versioned) archetype plugin

Archetype plugin provides:

core build settings (style, scala version, etc.)

archetype build settings (deploy, publish, etc.)

archetype dependencies (e.g. spray, akka, scopt)

Project upgraded upgraded by upgrading plugin version

Svc A

Plugins Projects Dependencies

akka 2.3.2

spray 1.3.0

dispatch 0.11.1Svc B

akka 2.3.4

spray-client 1.3.2

spray 1.3.1Svc C

scala 2.10.2

scala 2.11.1

akka 2.3.6

spray 1.3.2

Svc D

packager 0.6.0

scalastyle

packager 0.7.2

scalariform 1.3

scalariform 1.5

Web Services

web-service 1.0

Svc Aakka 2.3.2spray 1.3.0

dispatch 0.11.1

Svc B

Svc C

scala 2.10.2

Svc D

packager 0.6.0 scalariform 1.3

web-service 2.0

akka 2.3.4spray 1.3.1

dispatch 0.11.1

scala 2.11.1packager 0.6.0 scalariform 1.3

scalastyle

web-service 2.1

akka 2.3.6spray 1.3.2

spray-client 1.3.2

scala 2.11.1packager 0.7.2 scalariform 1.5

scalastyle

Plugins Projects DependenciesWeb Services

sbt-revolver

sbt-revolver

sbt-revolver

web-service 1.0 Svc A

Archetype Plugins Projects

Svc B

Svc C

Svc D

web-service 2.0

web-service 2.1