freeing yourself from an rdbms architecture
DESCRIPTION
Explore how we can begin to move functionality from a typical RDBMS application to one that uses tools and frameworks like MongoDB, Solr and Redis. At the end, the architecture we've evolved looks similar to.........TRANSCRIPT
![Page 1: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/1.jpg)
Being RDBMS FreeDAVID HOERSTER
2014
![Page 2: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/2.jpg)
About Me C# MVP (Since April 2011)
Sr. Director of Web Solutions at RGP
Conference Director for Pittsburgh TechFest
Co-Founder of BrainCredits (braincredits.com)
Past President of Pittsburgh .NET Users Group and organizer of recent Pittsburgh Code Camps and other Tech Events
Twitter - @DavidHoerster
Blog – http://geekswithblogs.net/DavidHoerster
Email – [email protected]
![Page 3: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/3.jpg)
Traditional Architecture Data persistence is central to application
Generally monolithic
Jack of all trades; master of none
![Page 4: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/4.jpg)
Traditional Architecture
Client Web Server
App Server
Data Repository
Data TablesSessionCache (?)FT Search
![Page 5: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/5.jpg)
What if… We could break some pieces out
◦ Flatten structures for querying◦ Highly efficient search services◦ Pub/sub hubs◦ Remote caching with excellent performance◦ Session management outside a DB for load balanced environments
How would app then be architected?
![Page 6: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/6.jpg)
Search How do you search?
◦ LIKE ‘%blah%’ ?◦ Dynamic SQL◦ Full-Text
LIKE and Dynamic SQL can be quick to create◦ Tough to maintain
Full-Text gives power◦ Limited in search options
![Page 7: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/7.jpg)
Search Number of search services out there like
◦ Lucene◦ Solr
Lucene is a search engine◦ Embed in apps◦ .NET port (Lucene.NET)
Solr is search service◦ Built on Lucene◦ Connect apps to it
![Page 8: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/8.jpg)
Searching with Solr Disconnected from your application
Search content via HTTP REST calls
Can use SolrNet as a client◦ https://github.com/mausch/SolrNet
Document-based
![Page 9: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/9.jpg)
Searching with Solr private readonly ISolrOperations<T> _solr; public SolrSearchProvider(ISolrOperations<T> solr) { _solr = solr; }
public IEnumerable<T> Query(String searchString) { var options = new QueryOptions() { Fields = new List<String> {"title", "body", "lastModified" }.ToArray(), Highlight = new HighlightingParameters() { BeforeTerm = "<strong><em>", AfterTerm = "</em></strong>", Fields = new List<String> { "title", "body" }.ToArray(), Fragsize = 100 } }; var results = _solr.Query(new SolrQuery(searchString), options); return results; }
![Page 10: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/10.jpg)
Evolving Architecture
Client Web Server
App Server
Data Repository
Data TablesSessionCache (?)
Search Service
Query
Write
![Page 11: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/11.jpg)
Data Storage Typically, RDBMS is the de facto standard
◦ SQL Server◦ MySQL◦ PostgreSQL◦ Oracle (Yikes!!)
But do you really need it?
![Page 12: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/12.jpg)
Data StorageGet all the orders for user ‘David’ in last 30 days
SELECT c.FirstName, c.MiddleName, c.LastName, soh.SalesOrderID, soh.OrderDate, sod.UnitPrice, sod.OrderQty, sod.LineTotal, p.Name as 'ProductName', p.Color, p.ProductNumber, pm.Name as 'ProductModel', pc.Name as 'ProductCategory', pcParent.Name as 'ProductParentCategory'FROM SalesLT.Customer c INNER JOIN SalesLT.SalesOrderHeader soh
ON c.CustomerID = soh.CustomerIDINNER JOIN SalesLT.SalesOrderDetail sod ON soh.SalesOrderID = sod.SalesOrderIDINNER JOIN SalesLT.Product p ON sod.ProductID = p.ProductIDINNER JOIN SalesLT.ProductModel pm ON p.ProductModelID = pm.ProductModelIDINNER JOIN SalesLT.ProductCategory pc ON p.ProductCategoryID = pc.ProductCategoryIDINNER JOIN SalesLT.ProductCategory pcParent ON pc.ParentProductCategoryID = pcParent.ProductCategoryID
WHERE c.FirstName = 'David'AND soh.OrderDate > (GETDATE()-30)
![Page 13: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/13.jpg)
Data Storage
Wouldn’t it be great if it were something like this?
SELECT FirstName, MiddleName, LastName, SalesOrderID, OrderDate, UnitPrice, OrderQty, LineTotal, ProductName, Color, ProductNumber, ProductModel, ProductCategory, ProductParentCategoryFROM CustomerSalesWHERE FirstName = 'David'
AND OrderDate > (GETDATE()-30)
![Page 14: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/14.jpg)
Data Storage Maybe a document database can be of use
Number out there◦ MongoDB◦ RavenDB◦ Couchbase
Flattened structures without relational ties to other collections
Essentially object databases
![Page 15: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/15.jpg)
Looking at MongoDB Server can have databases
Databases contain collections (like a table)
Collections contain documents (like rows)
Documents can be structured, have constraints, have primary key
![Page 16: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/16.jpg)
Working with Mongo’s C# Client public class MongoContext<T> : IContext<T> where T : class, new() { private IDictionary<String, String> _config; private readonly MongoCollection<T> _coll;
public MongoContext(IDictionary<String, String> config) { _config = config;
var client = new MongoClient(config["mongo.serverUrl"]); var server = client.GetServer(); var database = server.GetDatabase(config["mongo.database"]);
_coll = database.GetCollection<T>(config["mongo.collection"]); }
public IQueryable<T> Items { get { return _coll.FindAll().AsQueryable(); } } }
![Page 17: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/17.jpg)
Working with Mongo’s C# ClientEncapsulate my queries and commands
public class FindPageById : ICriteria<Page> { private readonly String _id; public FindPageById(String pageId) { _id = pageId; }
public IEnumerable<Page> Execute(IContext<Page> ctx) { return ctx.Items.Where(p => p.Id == _id).AsEnumerable(); } }
![Page 18: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/18.jpg)
Working with Mongo’s C# ClientInvoke my query/command public class TemplateController : IportalBaseController { private readonly IContext<Page> _pages;
public TemplateController(IContext<Page> ctx) : base() { _pages = ctx; }
[HttpGet] public async Task<IportalPageMetadata> Section(String cat, String page) { var id = String.Format("{0}/{1}", cat, page);
var thePage = new FindPageById(id) .Execute(_pages) .FirstOrDefault(); ... } }
![Page 19: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/19.jpg)
Working with Mongo’s C# ClientWriting to Mongo is just as simple...
[HttpPost]public async Task<Boolean> Post(Page page){ var userId = await GetUserId();
new CreatePage(page, userId) .Execute(_pages);
_searchPage.Insert(page);
return true;}
![Page 20: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/20.jpg)
Evolving Architecture
Client Web Server
App Server
Data Repository
Some data (?)SessionCache (?)
Search Service
Query
Write
Document Repository
Write
Query
![Page 21: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/21.jpg)
Session and Cache Data Generally short-lived for users
Fairly static for cached data
Key/value stores can serve us well here◦ Redis
Redis has two good .NET client libraries◦ StackExchange.Redis◦ ServiceStack.Redis
![Page 22: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/22.jpg)
Using Redis public class RedisSessionManager : ISessionManager { private static ConnectionMultiplexer _redis = null; private readonly IDictionary<String, String> _config;
public RedisSessionManager(IDictionary<String, String> config) { if (_redis == null) { _redis = ConnectionMultiplexer.Connect(config["session.serverUrl"].ToString()); } _config = config; } public async Task<Boolean> CreateSession(String portalId, String userId, String fullName) { var time = DateTime.UtcNow.ToString(); var timeout = _config.ContainsKey("session.timeout");
var vals = new HashEntry[] { new HashEntry("userid", userId), new HashEntry("login", time), new HashEntry("lastAction", time), new HashEntry("fullName", fullName) };
await RedisDatabase.HashSetAsync(portalId, vals); return await RedisDatabase.KeyExpireAsync(portalId, TimeSpan.FromMinutes(timeout)); } }
![Page 23: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/23.jpg)
Using Redis
public async Task<Boolean> ExtendSession(String portalId) { var timeout = _config.ContainsKey("session.timeout"); await RedisDatabase.HashSetAsync(portalId, "lastAction", DateTime.UtcNow.ToString()); return await RedisDatabase.KeyExpireAsync(portalId, TimeSpan.FromMinutes(timeout)); }
public async Task<Boolean> ExpireSession(String portalId) { return await RedisDatabase.KeyDeleteAsync(portalId); }
![Page 24: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/24.jpg)
Using Redis At login:
await Session.CreateSession(portalid, UserHandle, fullName);
Upon log out:
await Session.ExpireSession(portalCookie.Value);
![Page 25: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/25.jpg)
Evolving Architecture
Client Web Server
App Server
Data Repository
Some data (?)Search Service
Query
Write
Document Repository
Write
Query
Session/Cache
Service
![Page 26: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/26.jpg)
Why Data Store We’re left with a database with not much use
◦ Transactional data in document store◦ Search documents in Solr◦ Session, caching, etc. in key/value or caching service like Redis
What it probably ends up acting as is…
![Page 27: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/27.jpg)
Evolving Architecture
Client Web Server
App Server
Event Store 2-3 flat tablesEvent data
Search Service
Query
Write
Document Repository
Write
Query
Session/Cache
Service
Queue?
![Page 28: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/28.jpg)
(D)Evolved ArchitectureClient
Web Server
App Server
Event Store
Search Service
Query
Write
Doc Repo
Write
Query
Session/Cache
Service
Queue?
![Page 29: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/29.jpg)
(D)Evolved Architecture Pick and choose what components work best
Don’t use them just to use them
Proof-of-Concept / Prototype
![Page 30: Freeing Yourself from an RDBMS Architecture](https://reader035.vdocuments.site/reader035/viewer/2022081412/545ca678b0af9fa92c8b4966/html5/thumbnails/30.jpg)
Why look to be RDBMS free Searching
◦ More than just full-text needs
Data◦ Choose a system that you can model the business◦ Not the other way around
Caching / Session Values / PubSub◦ Offload necessary?◦ Ensure performance
Maintenance and support big factors to consider