composing project archetyps with sbt autoplugins

Composing Project Archetypes with SBT

AutoPluginsMark Schaake

allenai.org“contribute to humanity through high-impact AI research and engineering”

http://allenai.org

Combatting Multiple Build Maintenance Hell with Archetype AutoPlugins

Mark Schaake

allenai.org“contribute to humanity through high-impact AI research and engineering”

http://allenai.org

“When you have very limited visibility and control over many project builds to the point

where you feel paralyzed to maintaining cross-project consistency.”

(Multiple Build Maintenance Hell)MBMH

• visibility: what libraries, plugins, versions? • control: how to enforce team standards, upgrades? • consistency: minimize context switching

SOA, Inc.

• SOA architecture

• Scala, Akka, Spray

Svc A

Plugins Projects Dependencies

Svc A


packager 0.6.0

scalariform 1.3

// native packager for generating deployable artifacts addSbtPlugin( "com.typesafe.sbt" % "sbt-native-packager" % "0.6.0") // format our code in SBT instead of our editors addSbtPlugin( "com.typesafe.sbt" % "sbt-scalariform" % "1.3.0")

project/plugins.sbt

Svc A


packager 0.6.0

scalariform 1.3


project/plugins.sbt

akka 2.3.2

spray 1.3.0

dispatch 0.11.1

scala 2.10.2

organization := "com.soa" name := "service-a" scalaVersion := "2.10.2" libraryDependencies ++= Seq( "io.spray" % "spray-can" % "1.3.0", "io.spray" % "spray-routing" % "1.3.0", "com.typesafe.akka" %% "akka-actor" % "2.3.2", "com.typesafe.akka" %% "akka-slf4j" % "2.3.2", "net.databinder" %% "dispatch-core" % "0.11.1", "ch.qos.logback" % "logback-classic" % "1.1.2" ... )

build.sbt


// native packager for generating deployable artifacts addSbtPlugin( "com.typesafe.sbt" % "sbt-native-packager" % "0.6.0") // format our code in SBT instead of our editors addSbtPlugin( "com.typesafe.sbt" % "sbt-scalariform" % “1.3.0")

addSbtPlugin( “org.scalastyle" %% "scalastyle" % “0.5.0")

Svc A


akka 2.3.2

spray 1.3.0

dispatch 0.11.1Svc B

akka 2.3.4

spray 1.3.1

scala 2.10.2

scala 2.11.1

packager 0.6.0

scalastyle

scalariform 1.3

organization := "com.soa" name := "service-a" scalaVersion := "2.10.2" libraryDependencies ++= Seq( "io.spray" % "spray-can" % "1.3.0", "io.spray" % "spray-routing" % "1.3.0", "com.typesafe.akka" %% "akka-actor" % "2.3.2", "com.typesafe.akka" %% "akka-slf4j" % "2.3.2", "net.databinder" %% "dispatch-core" % "0.11.1", "ch.qos.logback" % "logback-classic" % "1.1.2" ... )

build.sbt

organization := "com.soa" name := "service-b" scalaVersion := "2.11.1" libraryDependencies ++= Seq( "io.spray" % "spray-can" % "1.3.1", "io.spray" % "spray-routing" % "1.3.1", "com.typesafe.akka" %% "akka-actor" % "2.3.4", "com.typesafe.akka" %% "akka-slf4j" % "2.3.4", "net.databinder" %% "dispatch-core" % "0.11.1", "ch.qos.logback" % "logback-classic" % "1.1.2" ... )

project/plugins.sbt

Svc A


akka 2.3.2

spray 1.3.0


akka 2.3.4

spray-client 1.3.2

spray 1.3.1Svc C

scala 2.10.2

scala 2.11.1

akka 2.3.6

spray 1.3.2

packager 0.6.0

scalastyle

packager 0.7.2

scalariform 1.3

scalariform 1.5

Svc A


akka 2.3.2

spray 1.3.0


akka 2.3.4

spray-client 1.3.2

spray 1.3.1Svc C

scala 2.10.2

scala 2.11.1

akka 2.3.6

spray 1.3.2

Svc D

packager 0.6.0

scalastyle

packager 0.7.2

scalariform 1.3

scalariform 1.5

Svc A


akka 2.3.2

spray 1.3.0


akka 2.3.4

spray-client 1.3.2

spray 1.3.1Svc C

scala 2.10.2

scala 2.11.1

akka 2.3.6

spray 1.3.2

Svc D

packager 0.6.0

scalastyle

packager 0.7.2

scalariform 1.3

scalariform 1.5…

Lib X

Lib Y

CLI Z scopt…

sbt-release

assembly

project/plugins.sbt 6

build.sbt 50

Project build LOC 56

x 26 projects

= 1,456 LOC!

…

…

Avg Project Build LOC

and mostly boilerplate

project/plugins.sbt 6

build.sbt 50

Project build LOC 56

x 26 projects

= 1,456 LOC!

…

…

Avg Project Build LOC

and mostly boilerplateMBMH!

and MBMH

Build

LO

C

500

1,000

1,500

2,000

2,500

Proj

ects

15

30

45

60

Jan March May July Sept Nov

Projects Build LOC

and MBMH

Build

LO

C

500

1,000

1,500

2,000

2,500

Proj

ects

15

30

45

60


Projects Build LOC

M

Build

LO

C

500

1,000

1,500

2,000

2,500

Proj

ects

15

30

45

60


Projects Build LOC

MBMH

and MBMH

Yay!

Solving MBMH

Solving• Maximize consistency across projects

MBMH


• Minimize build complexity (LOC)

MBMH



• Maximize agility to evolve standards

MBMH




• Allow for stragglers (don’t force upgrades)

MBMH




• Allow for stragglers (don’t force upgrades)

• Easy build upgrade path

MBMH

Other Considerations


• Archetype settings shared by similar projects



• Core settings common to all projects




• Formatting and Style




• Formatting and Style

• Generate Git version resource

Solution: Archetype SBT Plugins


• Projects enable a single (versioned) archetype plugin



• Archetype plugin provides:

• core build settings (style, scala version, etc.)





• archetype build settings (deploy, publish, etc.)






• archetype dependencies (e.g. spray, akka, scopt)






• archetype dependencies (e.g. spray, akka, scopt)

• Project upgraded upgraded by upgrading plugin version

Svc A


akka 2.3.2

spray 1.3.0


akka 2.3.4

spray-client 1.3.2

spray 1.3.1Svc C

scala 2.10.2

scala 2.11.1

akka 2.3.6

spray 1.3.2

Svc D

packager 0.6.0

scalastyle

packager 0.7.2

scalariform 1.3

scalariform 1.5

Web Services

web-service 1.0

Svc Aakka 2.3.2

spray 1.3.0

dispatch 0.11.1

Svc B

Svc C

scala 2.10.2

Svc D

packager 0.6.0

scalariform 1.3

web-service 2.0

akka 2.3.4

spray 1.3.1

dispatch 0.11.1

scala 2.11.1packager 0.6.0

scalariform 1.3

scalastyle

web-service 2.1

akka 2.3.6

spray 1.3.2

spray-client 1.3.2

scala 2.11.1packager 0.7.2

scalariform 1.5

scalastyle

<app-specific dependencies>

Plugins Projects DependenciesWeb Services

sbt-revolver

sbt-revolver

sbt-revolver

web-service 1.0 Svc A

Archetype Plugins Projects

Svc B

Svc C

Svc D

web-service 2.0

web-service 2.1

library 1.0

library 1.1

cli 1.0

Lib X

Lib Y

CLI Z

…

…

…

web-app 1.0 App W…

Before Archetypes

import com.typesafe.sbt.SbtNativePackager._import org.scalastyle.sbt.ScalastylePlugin

scalaVersion := "2.10.2"

name := "Proj B"

Revolver.settings

scalariformSettings

ScalariformKeys.preferences := ScalariformKeys.preferences.value .setPreference(DoubleIndentClassDeclaration, true) .setPreference(MultilineScaladocCommentsStartOnFirstLine, true) .setPreference(PlaceScaladocAsterisksBeneathSecondAsterisk, true)

ScalastylePlugin.Settings

libraryDependencies ++= Seq( "com.typesafe.akka" %% "akka-actor" % "2.3.1", "io.spray" % "spray-can" % "1.3.2", "io.spray" % "spray-routing" % "1.3.2", "io.spray" % "spray-cace" % "1.3.2", "io.spray" % "spray-json" % "1.2.6", "net.databinder.dispatch" %% "dispatch-core" % "0.11.0")…

addSbtPlugin("com.github.gzeitz" % "sbt-release" % "0.8")

addSbtPlugin("com.typesafe.sbt" % "sbt-scalariform" % "1.2.1")

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "0.6.4")

addSbtPlugin("io.spray" % "sbt-revolver" % "0.6.2")

addSbtPlugin("org.allenai.plugins" % "version-injector" % "0.2.2")

addSbtPlugin("org.allenai.plugins" % "publisher" % "0.2.2")

Before Archetypes

import com.typesafe.sbt.SbtNativePackager._import org.scalastyle.sbt.ScalastylePlugin

scalaVersion := "2.10.2"

name := "Proj B"

Revolver.settings

scalariformSettings

ScalariformKeys.preferences := ScalariformKeys.preferences.value .setPreference(DoubleIndentClassDeclaration, true) .setPreference(MultilineScaladocCommentsStartOnFirstLine, true) .setPreference(PlaceScaladocAsterisksBeneathSecondAsterisk, true)

ScalastylePlugin.Settings

libraryDependencies ++= Seq( "com.typesafe.akka" %% "akka-actor" % "2.3.1", "io.spray" % "spray-can" % "1.3.2", "io.spray" % "spray-routing" % "1.3.2", "io.spray" % "spray-cace" % "1.3.2", "io.spray" % "spray-json" % "1.2.6", "net.databinder.dispatch" %% "dispatch-core" % "0.11.0")…

addSbtPlugin("com.github.gzeitz" % "sbt-release" % "0.8")

addSbtPlugin("com.typesafe.sbt" % "sbt-scalariform" % "1.2.1")

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "0.6.4")

addSbtPlugin("io.spray" % "sbt-revolver" % "0.6.2")

addSbtPlugin("org.allenai.plugins" % "version-injector" % "0.2.2")

addSbtPlugin("org.allenai.plugins" % "publisher" % "0.2.2")

name := "Proj B"

enablePlugins(WebServicePlugin)

addSbtPlugin("org.allenai.plugins" % "plugins" % “1.0”)

After Archetypes

LOC Before 56LOC After 3

LOC Reduction 95%

Implementing Archetype Plugins


• How to wrap / depend on other plugins?


• How to wrap / depend on other plugins?

• How to include core settings in each archetype?

Arch

etyp

e Pl

ugin

s

What AI2 NeededLibrary Web Service

libraryDependencies:- akka- spray- spray-json- typesafeConfig- ai2Common- ai2Testkit

wrapped plugins:- sbt-revolver

Web ApplibraryDependencies:- ai2Webapp

settings:- npm:*- reStart dependsOn npm:build- universal:mappings + npm:target

CLIlibraryDependencies:- scopt

Arch

etyp

e Pl

ugin

s

What AI2 NeededLibrary Web Service






Auxi

liary

Plu

gins

Core SettingsscalaVersion: 2.10.4scalacOptions: …conflictManager: strict

common library ModuleIDs- ai2Common, ai2Testkit, …- akka, spray, spray-json

Deploywrapped plugins:- sbt-native-packager

command:- deploy

Versioningtasks:- injectVersion- injectArtifact- injectGit- gitCommitDate- gitRemotes- gitSha1- gitDescribe

Releasewrapped plugins:- sbt-release

settings:- custom versioning

NodeJscommands:- npm <args>- npm:install- npm:build- npm:test- nom:clean

settings:- npm:nodeProjectDir- npm:nodeProjectTarget- npm:buildEnvironment- nom:environment

Stylewrapped plugins:- scalariform- scalastyle

settings:- apply scalariform- apply scalastyle

(Vanilla) SBT Plugins?

• Loosely defined API

• What are the right conventions?

Adding Popular SBT Plugin Settings

“To add the default plugin settings, add the following to your build.sbt…”

Adding Popular SBT Plugin Settings

“To add the default plugin settings, add the following to your build.sbt…”

import com.typesafe.sbt.SbtNativePackager._ import NativePackagerKeys._

packageArchetype.java_application

sbt-native-packager

org.scalastyle.sbt.ScalastylePlugin.Settings

sbt-scalastyle

scalariformSettings

sbt-scalariform

Revolver.settings

sbt-revolver

SBT AutoPlugins

• The new Plugin standard (since SBT 0.13.5)

SBT AutoPlugins


• Well-defined plugin API

SBT AutoPlugins



• Less need for conventions

SBT AutoPlugins



• Less need for conventions

• Killer feature: can compose plugins via `requires`

abstract class AutoPlugin { … /** When this AutoPlugin is enabled, all required * plugins will also be enabled automatically prior * to enabling this AutoPlugin. */ def requires: Plugins = empty

/** The [[Setting]]s to add in the scope of each * project that activates this AutoPlugin. */ def projectSettings: Seq[Setting[_]] = Nil … }

SBT AutoPlugins

https://github.com/sbt/sbt/blob/0.13.7/main/src/main/scala/sbt/Plugins.scala

https://github.com/sbt/sbt/blob/0.13.7/main/src/main/scala/sbt/Plugins.scala

Core Settings AutoPluginobject CoreSettingsPlugin extends AutoPlugin { override def requires = StylePlugin && VersionInjectorPlugin override def projectSettings = Seq( scalaVersion := CoreDependencies.defaultScalaVersion, scalacOptions ++= Seq("-target:jvm-1.7", "-Xlint", "-deprecation"), javacOptions ++= Seq("-source", "1.7", "-target", “1.7"), … ) }






Core Settings AutoPlugin

When CoreSettingsPlugin is enabled…

object CoreSettingsPlugin extends AutoPlugin { override def requires = StylePlugin && VersionInjectorPlugin override def projectSettings = Seq( scalaVersion := CoreDependencies.defaultScalaVersion, scalacOptions ++= Seq("-target:jvm-1.7", "-Xlint", "-deprecation"), javacOptions ++= Seq("-source", "1.7", "-target", “1.7"), … ) }


1. adds projectSettings from StylePlugin and VersionInjector




1. adds projectSettings from StylePlugin and VersionInjector2. adds projectSettings from CoreSettings



Composing an Archetype Plugin




command:- deploy


Web ServicelibraryDependencies:- akka- spray- spray-json- typesafeConfig- ai2Common- ai2Testkit




object WebServicePlugin extends AutoPlugin { override def requires = CoreSettingsPlugin && DeployPlugin override def projectSettings = Revolver.settings ++ Seq( libraryDependencies ++= Seq( akkaActor, akkaLogging, sprayCan, sprayRouting, sprayJson, allenAiCommon, allenAiTestkit % "test")) }





command:- deploy







When WebServicePlugin is enabled…


1. Style and Versioning applied




command:- deploy









1. Style and Versioning applied2. CoreSettings and Deploy applied




command:- deploy









1. Style and Versioning applied2. CoreSettings and Deploy applied3. WebService applied




command:- deploy








Arch

etyp

e Pl

ugin

sAu

xilia

ry P

lugi

nsWeb Service









command:- deploy

Arch

etyp

e Pl

ugin

sAu

xilia

ry P

lugi

nsWeb Service










Library


command:- deploy

Arch

etyp

e Pl

ugin

sAu

xilia

ry P

lugi

nsWeb Service











Library


command:- deploy

Arch

etyp

e Pl

ugin

sAu

xilia

ry P

lugi

nsWeb Service











Library


command:- deploy



NodeJscommands:- npm <args>- npm:install- npm:build- npm:test- nom:clean

settings:- npm:nodeProjectDir- npm:nodeProjectTarget- npm:buildEnvironment- nom:environment

Archetype Plugins in Action

Resources

• SBT docs - http://www.scala-sbt.org/0.13/docs/Plugins.html

• Tutorial - http://mukis.de/pages/sbt-autoplugins-tutorial/

• AI2 sbt-plugins: https://github.com/allenai/sbt-plugins

Questions

Mark Schaake [email protected]

@markschaake

Project Archetype Recipe• Create a company “sbt-plugins” project

• Wrap third-party plugins (scalariform, scalastyle, etc.)

• Define a core settings AutoPlugin that enforces team standards

• Define archetype plugins that require core settings and add in archetype-specific settings

• Publish to Bintray and serve!

composing project archetyps with sbt autoplugins

Software

native packager

editors addsbtplugin

scalastyle scalariform

addsbtplugin org

seq io

highimpact ai research

project builds

crossproject consistency