modularizing your grails application with private plugins - springone 2gx 2012

Modularizing Your Grails Application

with Private PluginsKen Liu

IEEESpringOne 2GX 2012

17 Oct 2012

Wednesday, October 17, 12

About Me

• Web development with Java since JDK 1.0.2 (applets, anyone?)

• Groovy/Grails since 2009

• Lead developer for IEEE Spectrum

• Always been a closet fan of dynamic languages (starting with Perl)

About IEEE• Institute of Electrical and Electronics

Engineers

• Widely known for standards - 802.11

• World’s largest technology professional association - 400,000 engineers in 160 countries

• Publishes nearly a third of the world’s technical literature in EE/CS

About IEEE Spectrum• Flagship publication of IEEE

• General interest technology magazine with focus on EE topics

• 2012 National Magazine Award winner

IEEE Spectrum website

• http://spectrum.ieee.org

• ~1.5M PV/mo

• Frequent referrals from HN, Reddit, /.

• 100% Groovy/Grails since 2009 - both front-end and CMS

• Akamai/Weblogic/Oracle stack

• Award-winning coverage of ROBOTS!

Agenda

• Grails plugins overview

• Grails plugin development basics

• Application architecture with private plugins

• Lessons learned

To fork or not to fork?

• “We need a new website...”

• “You must use the standard corporate Enterprise CMS system.”

• “Can’t you guys just make a copy of the existing code and tweak it?”

• DANGER!

From “big ball of mud” to plugin architecture

Front EndControllers/Views/

Services(e.g. article, blog pages)

Domain classes(e.g Article,

BlogPost)

CMS Controllers/

Views/Services

Quartz Jobs(e.g. sitemap generation)

Spectrum TaglibUtility classes

(src/**)UrlMappings

Test classesFront End & CMSimages, CSS, JS

Where do we go from here?

• Lessons learned

Agenda

Plugin systems and component models

• Plugin systems are now commonplace

• Firefox, Eclipse, Jenkins, JEdit

• Package managers, OSGi, Ruby on Rails Engines

• Component-oriented software?

• Coarse-grained high-level reuse, separation of concerns

Grails plugins in a nutshell

• Tightly integrated into Grails system

• Can implement common functionality in plugins that is overridden by application

• Can create new Artifact types

• Interact with Spring context

• Integrated with build system

Grails plugins in a nutshell

• Central public plugin repository

• Version management

• Dependency management

Grails - plugins inside

• Many parts of Grails itself are built as plugins

• Some default plugins can even be swapped out

• Tomcat -> Jetty

• GORM/Hibernate -> Redis GORM

Public Grails plugins• 850 public plugins in the central

plugins repository

• Search on plugin portalhttp://grails.org/plugins/

• Quickly bootstrap new applications

• Spend some time nowsave some time later

• Follow @grailsplugins for updates

Selected Grails plugins

Spring Security - user authenticationCache - caching using annotationsQuartz - job scheduling (like cron)DB Migration - controlled schema changesMail - send email with JavaMailResources - web performance optimizationConsole - in-app interactive consoleCodeNarc - static code analysisSpock - enable Spock unit testing framework

What can go into a plugin?

• Really, anything

• Standard Grails artifacts: GSPs, Controllers, Services, i18n, TagLibs - just like in app

• Define custom Grails artifact types: see Quartz plugin “Jobs”

• Plugins can depend on other plugins

Agenda

• Lessons learned

Creating a new plugin

grails create-plugin myplugin

Creating a new plugin

Same structure as a Grails application, but with an extra plugin descriptor file

Plugin Descriptorclass MypluginGrailsPlugin { // the plugin version def version = "0.1"

// the version or versions of Grails the plugin is designed for def grailsVersion = "2.1 > *"

// the other plugins this plugin depends on def dependsOn = [:]

def doWithDynamicMethods = { ctx -> // TODO Implement registering dynamic methods to classes (optional) }

def doWithApplicationContext = { applicationContext -> // TODO Implement post initialization spring config (optional) }

Excluded plugin files

Grails excludes certain files from packaging

/grails-app/conf/BootStrap.groovy/grails-app/conf/BuildConfig.groovy/grails-app/conf/Config.groovy/grails-app/conf/DataSource.groovy(and any other *DataSource.groovy)/grails-app/conf/UrlMappings.groovy/grails-app/conf/spring/resources.groovy/web-app/WEB-INF/**/web-app/plugins/**/test/**

Running a plugin

• A plugin is a Grails app!

• grails run-app from plugin dir

• excluded files loaded during run-app

• useful for testing (test-app works too)

Packaging a plugin

• grails package-plugin

• creates a zip file that can be installed & distributed

Installing a plugin

from local directory:grails install-plugin /path/to/grails-example-0.1.zip

from URL: grails install-plugin http://myserver.com/plugins/grails-example-0.1.zip

in BuildConfig.groovy (“in-place” plugin):// Uncomment this only for local development - do not check in!grails.plugin.location.'spectrum-core' = '../spectrum-core'

Plugin Development

• Easy: create Controllers, Services, Views, Taglibs, etc.

• Specify resource paths <g:resource>

• Can alter Spring context

• Hook into build system, reload events

• Lessons learned

Agenda

Public vs. Private plugins

• Highly cohesive

• Maximize reuse

• Sometimes provide “horizontal slices” - e.g. Spring security core

• Most don’t depend on other plugins

Public plugins

Public vs. Private plugins

• Proprietary code

• Application or domain-specific

• Code reuse within an organization

• Can be coarse-grained, heavyweight

• Private forks of public plugins

• Enable interesting options for application architecture

Private Plugins

• Grails applications tend to become unmanageable over time

• Convention-based directory trees(e.g. /grails-app/views )

• Rapid development/scaffolding

• Need “separation of concerns”

Front EndControllers/Views/

Services(e.g. article, blog pages)

Domain classes(e.g Article,

BlogPost)

CMS Controllers/

Views/Services

Quartz Jobs(e.g. sitemap generation)

Spectrum TaglibUtility classes

(src/**)UrlMappings

Test classesFront End & CMSimages, CSS, JS

Spectrum application before refactoring

Reusable domain model

• Reuse a set ofDomain classes between applications

• Different applications, same DB instance

• Hibernate not included in plugins by default

• Consider Hibernate caching issues

Primary webapplication

Domain Model plugin

API endpoint application

Domain Model plugin

Reporting application

Domain Model plugin

Refactored with plugin

Front EndControllers/Views

Core Domain classes

CMS Controllers/

Views/Services

Quartz Jobs

Front End Taglib

Utility classes(src/**)

Front EndUrlMappings

Test classes

Front Endimages, CSS, JS

CMS test classes

CoreUrlMappings

Spectrum Core PluginSpectrum Application

CMSimages, CSS, JS

Common Front End Controllers

Core Services

CMS Taglib

Depends on other plugins: spring-security-core, quartz, searchable, ckeditor, and others

Different web sites, common backend code

spectrum-corePlugin

Spectrumwebsite

The Institutewebsite

Same Controller, different Views

FrontendArticleController#show()

Spectrum applicationshow.gsp

The Institute applicationshow.gsp

Same Controller, different Views

• Define common Controller in plugin

• View resolution: application first, then plugin

• Can specify plugin using<g:render plugin=”myplugin”>

• The promise of MVC fulfilled!

Plugin UrlMappings• UrlMappings.groovy excluded from plugin

• *UrlMappings.groovy merged into application mappings

• Application can override plugin mappings

• Enables “mini application” in plugin

• Can’t remove unwanted mappings, only override

Config.groovy• Config.groovy is excluded from plugin

• Might want to fix certain config properties - won’t need them in application Config

• Manually merge config in doWithSpring

• application.config.merge(new ConfigSlurper().parse(application.classLoader.loadClass('MyPluginConfig')))

Extending plugindomain model

Domain classes in plugins can be

extended in application

Content classDomain model plugin

Webinar extends ContentApplication

Publishing private plugins

• Publish to internal corporate Maven repo - e.g. Artifactory or Nexus

• Release plugin - installed by default

• Old-style svn-based repos supported, but not recommended for Grails 2

• Customize plugin metadata with internal Maven groupId

• Publish from Jenkins

Declaring private plugin dependencies

repositories { grailsPlugins() /* other default repos here */

/* point to internal corporate Maven repo */ mavenRepo 'http://mvn.ieee.org/nexus/content/repositories/spectrum/'}

plugins { /* other plugin dependencies here */ compile 'org.ieee.spectrum.plugins:spectrum-core:2.0-SNAPSHOT'}

BuildConfig.groovy (Maven/Ivy dependencies)

Day-to-day development

• Use “in-place” plugin (exploded plugin) in your local dev workspace (BuildConfig.groovy)

• Private plugins tend to be updated along with application features

• Beware of potential breaking changes(in other applications)

• Define deprecation strategy and use @deprecated javadoc

Refactoring to plugin

• Determine candidates for refactoring

• Create new empty plugin & plugin descriptor

• Move Grails artifacts (e.g. Controllers, Services, Views, etc.) into new plugin

• Move dependent files from /src

• Move related test files into new plugin• Remove hard coded paths to resources

• Run your functional test suite

Refactoring to plugin

• Start small - don’t create a whole bunch of plugins from the outset

• Consider migrating version history

Super-project with pluginsOne application with several in-place plugins

/myapplication /grails-app ... /pluginA /grails-app /pluginB /pluginC

grails.plugin.location.pluginA = 'pluginA' // path to plugingrails.plugin.location.pluginB = 'pluginB'grails.plugin.location.pluginC = 'pluginC'

In BuildConfig.groovy:

Public website,private back end

Deploy public and private versions of application with shared code

Business-facing

application

Core application

plugin

Public-facingapplication

Core application

plugin

PublicInternet

Private intranet

Vertical slices / mini Grails applications

• Plugins can be self contained

• UrlMappings + Controllers + Domains + Views + Services = Grails application

• Remove views -> “white label” application

• Embed a small application within another application

• Vertically slice your application into plugins

Plugins as a platform

Clinical Data Management Platform plugin

Clinical data management server

application

Clinical data review application

Data Captureplugin

Data Cleaningplugin

Data Validationplugin

Super-plugin to combine plugin

dependencies into a platform or

domain-specific framework

Sensitive code protection

Data Ingestionplugin

Data Analysisplugin

“secret sauce”

Full application

Shell application (external team)

Split applicationinto separate parts to isolate key algorithms from external vendor

• Lessons learned

Agenda

Domain/Schema management

• Changes to domain model need to be carefully managed

• Avoid dbCreate=update

• Adopt DDL file naming convention:spectrum-core-2.1.sqlspectrum-core-2.2.sql

• Bootstrap-triggered migrations

• DB Migrations plugin (Liquibase)

Config.groovy changes over time

• Easy to add a new entry to Config.groovy and then forget about it

• Group together plugin config properties into a closure

• Don’t forget about external config files and differences in various environments

Maven/Ivy/build issues

• Maven/Ivy hell > JAR hell

• SNAPSHOT resolution error-prone

• See JIRA for bugs related to plugins

• Avoid creating many small plugins

• Strange problems? Look at your build.

Plugin versioning and branching

• Define version numbering strategy

• Be prepared to branch/merge

• Be sure to update plugin versions and tag your code consistently

• Just use Git - merging & cherry-picking are sweet

Code Rigidity

• Inflexibility to change/afferent coupling

• Avoid breaking changes (duh)

• Solution: automate your tests, catch breaking changes early

Continuous Integration

• Use Jenkins to trigger builds of applications that depend on your private plugin

• Run Plugin test suite with plugin build

• Publish snapshots to Maven repo from Jenkins

Plugin namespace conflicts

• Plugin artifacts all share same namespace with each other

• Potential for class name collisions, especially with “conventions”

• Planned enhancement for Grails 2.2

Plugin resources

• Plugin static resources (images, CSS, JS, etc.) packaged in a path corresponding to plugin name & version - /plugins/myplugin-1.0/*

• path changes between dev mode and WAR - <g:resource> handles correctly

• May need to redeploy with each plugin version bump

• Don’t accidentally share sensitive files!

Component interfaces

• Grails plugins don’t have any kind of external “interface” or contract besides Plugin Metadata

• Up to you to make the rules(not a bad thing)

Release strategy for 2013 website redesign

Success!

spectrum-core Plugin2.0, 2.1, 2.2

Spectrumwebsite

10.8, 10.9, 10.10

The Institutewebsite

Spectrumwebsite

4.0 - 2013 release

Takeaway

• Plugins are easy to create and use

• Private plugins are a safe, robust way to modularize Grails applications

• Private plugins enable coarse-grained reuse of application components

• Configuration management and OO principles are important (as always)

Additional resources

• @grailsplugins - Grails plugin updates - https://twitter.com/grailsplugins

• Grails reference docs chapter on plugins - http://grails.org/doc/latest/guide/plugins.html

• Inside Grails: The Build System and Plugin Management - http://www.slideshare.net/skillsmatter/grails-internals-ggug-dec-2009

• GGUG: Grails Plugins - Lessons to Learn http://skillsmatter.com/podcast/java-jee/grails-plugins

Questions?

Thanks!

feedback is appreciated.

@kenliu

k.a.liu@ieee.org

modularizing your grails application with private plugins - springone 2gx 2012

grails plugins

plugin grails packageplugin

grails system

public grails plugins

versions of grails

parts of grails

plugin version

groovy grailsappconfbuildcong

Technology

modularizing crosscutting concerns in software nalin saigal

springone 2gx - spring boot for devops

devops for normals, springone platform 2016

practical groovy domain-specific languages - springone...

springone 2015 報告会 - lattice + spring cloud netflix

springone europe 2009: time to live

springone 2gx 2015 - fullstack groovy developer

performance tuning grails applications springone 2gx 2014

springone 2gx 2011 - writing applications for cloud foundry...

zuul @ netflix springone platform

structuring and modularizing the network

modularizing crosscutting concerns with ptolemy

modularizing flex cairngorm and modules

structuring and modularizing the network -...

spring tooling update - new & noteworty (at springone 2011)

modularizing your configuration files for team development

apache kafka event-streaming platform for .net...

servicification: modularizing...

designing your own domain-specific language in groovy by...

modularizing content