MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing
Christian Brensing
Bayerisches Landesamt für Statistik und Datenverarbeitung
IuK / Rechenzentrum Süd
• Template-Processor (~ JSP)
• ODF and RTF
• PDF / PostScript postprocessing
• Groovy, Python, Ruby, Tcl
• In Production using RDBMS since 2008
• Central HR system (SAP) uses BayText to create output
Client for developing the templates (BayText-IDE)
ODF-Template using Ruby
Generated document
2 MillionGenerated documents per year
Tiny dataset50,000 records and 10 GB BLOB
a small system after all
So why migrate?
Because we can
developers.pop()Smaller teams
Easy to learnRead one book and you're done
Simple mapping
nodes
groups
templates
document_usages
folder_configurations
codes
permissions
requests
documents
folders
structures template_tags
roles
roles_roles
bundles
componentsdynafields
scripts
nodes
roles
properties
GridFS
{! "_id" : ObjectId(),! "_type" : "Document",! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Document",! "description" : "",! "language" : "ruby",! "format" : "odf",! "autoload_uplevel" : -1,! "requests" : [! {! "name" : "Test",! "description" : "foo", ! "xml" : zlib("<?xml ...?>")! }, ! ... ! ],! "structure" : [! { ! "alias" : "some alias",! "component_path" : "foo.bar.Document$Document",! "parent", -1 ! },! ... ! ], ! } ! "tags" : ["foo", "bar", "baz"]!}
public class RoleReadConverter implements Converter<DBObject, Role> {! @Override! public Role convert(DBObject source) {! Role role = new Role();! role.setName((String) source.get("name"));! ...! return role;! }!}!!!public class RoleWriteConverter implements Converter<Role, DBObject> {! @Override! public DBObject convert(Role source) {! return new BasicDBObject()! .append("name", role.getName())! .append(...);! }!}!!!// Convert a DBObject to an Entity!conversionService.convert(collection.findOne(...));
Easy mapping using Spring-ConversionService
Performanceup to 10x
ORM no moreThe MongoDB driver is all you need
Obstacles?
Mentality
Referential IntegrityMulti-Document-Update on denormalized schema
{! "_id" : ObjectId(),! "_type" : "Folder", ! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Folder",! "description" : "bla",! "bundles" : [! ... ! ]! "document_usages" : [! {! "name" : "foo",! "description" : "bla", ! "exec_order" : 5, ! "print_copies" : 2,! "type": "FAIR_COPY",! "document_path" : "a.b.Document",! "bundle" : "bar"! },! ...!}
Linking documents via a path instead of ObjectId
But paths are mutable!All references must be updated
Approach
• Pseudo-Transaction with an update locking mechanism in a custom oplog collection
• Periodical repair jobs finishing failed operations • Extended interpretation of eventual consistency
Unit-TestsContinuous Integration
F.I.R.S.T.
IsolatedTests are using a unique DB per JVM
@Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {! @Override! public void init() {! setTestProperties();! ! super.init();! ! // Delete test db! db().dropDatabase();! }! ! private void setTestProperties() {! String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());! String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");! String port = env.getProperty("db.port", "27016");! ! cmProperties.setProperty("servers", String.format("%s:%s", host, port));! cmProperties.setProperty("name", dbname);! }!! // OS-Username without prefix (e.g. maggie instead of lfstad-maggie)! private static String getSimpleUsername() {! String username = SystemUtils.USER_NAME;! int indexOfDash = username.indexOf('-');! return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;! }!}
Test-ApplicationContext
RepeatableCollections are dropped before each test method
public class MongoTestExcecutionListener extends AbstractTestExecutionListener {! @Override! public void beforeTestMethod(TestContext testContext) throws Exception {! purgeCollections();! }! ! private void purgeCollections() {! DB db = MongoDBHolder.getDB();! for (String collectionName : db.getCollectionNames()) {! if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {! continue;! }! DBCollection collection = db.getCollection(collectionName);! if (!collection.isCapped()) {! collection.drop();! }! }! }!}
Spring-TestExecutionListener
@RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration.class)!@TestExecutionListeners({! DependencyInjectionTestExecutionListener.class,! DirtiesContextTestExecutionListener.class,! MongoTestExcecutionListener.class!})!public abstract class MongoTestSupport {!}!!!!public class DocumentRepositoryTest extends MongoTestSupport {!}!
Test base class
95% Coverage
Summary
• Developing the data layer is fun again • Reduced complexity • Flat learning curve compared to SQL/ORM • use_mongo() unless transactions_required
Thank you!