open source: top issues in the top enterprise packages

1© 2016 Rogue Wave Software, Inc. All Rights Reserved.

1

Top open source lessonsfor every enterpriseEpisode 4:

Top issues in the top enterprise packages


2

Bill CrowellEnterprise architect, OpenLogic supportRogue Wave Software

Vince CoxEnterprise architect, OpenLogic supportRogue Wave Software

Presenters


3

Poll #1What percentage of your mission critical software is open source?

A: 0 to 25%B: 26 to 50%C: 51 to 75%

D: 75%


4

1. Introduction2. Setting the context3. Top issues: middleware + runtimes4. Top issues: databases5. Top issues: security6. Conclusion7. Q&A

Agenda


5

Who wrote LevelDB and what version of ActiveMQ did it debut in?

What percentage of web server market share does Apache HTTP Server hold?

Pop quiz


6

Introduction

What problems do our clients commonly run into?

From our experience…• ActiveMQ High Availability Shared Databases Using NFS• PostgreSQL Database Performance• Request Header Vulnerabilities


7

Poll #2What type of ActiveMQ persistence store do you use?

A: KahaDBB: LevelDB

C: Replicated LevelDBD: We don’t use message persistence with ActiveMQ

E: We don’t use ActiveMQ


8

Top issues:middleware +

runtimes


9

Scenario #1

Observing crashing or unresponsive broker

Typical configuration/requirements• Master/slave network broker setup with NFS mount• NFSv4 (SoftNAS) cloud-based shared file system available

with Amazon Web Services EC2 or GlusterFS• Often a high-throughput requirement (10k messages/second

of 1-3kb message size)

One of the following problems…• Master dies resulting in “no master”

scenario• Slave prematurely claims lock resulting in 2

master brokers leading to message loss• Continuous master/slave re-election• Increasing CPUs/memory doesn’t help• Non-existent prior to production


10

Scenario #1 solution

Use SAN, Replicated LevelDB, or Pluggable Storage Lockers

Three solutions• Use block-level iSCSI driver with Storage Area Network (SAN)• Master-slave for HA and Replicated LevelDB managed by

Zookeeper• Pluggable Storage Lockers

More points• Inherent flaws in OS-level filesystem locking mechanism• Exclusive file locks work great with a SAN but is most

expensive• Replicated LevelDB requires a more configuration and a

quorum of nodes (replicas / 2 + 1)• Pluggable Storage Locker/Lease Database Locker


11


Lease Database Locker

Points• Master must renew lease before lease expires• The lease period can be configured• If not renewed, then the slave takes ownership of the

lease becoming the new master

More points• Leased locks can survive database replica failovers• Can be used with any JDBC-compliant database• Make sure to uniquely name your brokers• Keep master/slave clocks synchronized with NTP

service• Uniquely name your brokers. Use connection pooling


12

Poll #3Do you use PostgreSQL?

A: We don’t use PostgreSQLB: We don’t use PostgreSQL but plan on using it in the future

C: We use PostgreSQL as a mission-critical application databaseD: We use PostgreSQL as part of another open source project

E: We are migrating away from PostgreSQL


13

Top issues:databases


14

Scenario #2

Database performance is poor

Symptoms• Sorting and querying take a long time• One particular query or web page is hanging the

database• “Sorry, too many clients already” or connection pool

is full

Where do I start?• Localize if possible: Data center, network,

database or application server?• When did it start• What changes took place?


15


Identify and analyzeQuick checks• top with ‘c’ command shows process ID, CPU, and

memory utilization• ”iostat –x –m 5” reveals disk IO wait times• ELK = Elasticsearch + Logstash + Kibana• “EXPLAIN ANALYZE <SQL>” shows execution time and

table scans

Tools• psql: SELECT pid, datname, usename, query FROM pg_stat_activity;

pid datname usename query42102 jboss jboss SELECT pid, datname, usename,

query FROM pg_stat_activity;

42103 jboss jboss SELECT video FROM news where...;


16


Identify and analyze

Tools• pg_stat_statmentswww.postgresql.org/docs/current/static/pgstatstatements.html

Note: pg_stat_statements requires more shared memory

postgresql.conf:shared_preload_libraries = ‘pg_stat_statements’

Server restart is required after enabling the shared library

Reset statistics: select pg_stat_reset();

https://www.postgresql.org/docs/current/static/pgstatstatements.html


17



ToolsMore on PostgreSQL performance: www.craigkerstiens.com/2013/01/10/more-on-postgres-performance/

SELECT (total_time / 1000 / 60) as total_minutes, (total_time/calls) as average_time, query FROM pg_stat_statements ORDER BY 1 DESC LIMIT 100;

The QueryTotal Query Time (in minutes)Average Time (in milliseconds)

http://www.craigkerstiens.com/2013/01/10/more-on-postgres-performance/

http://www.craigkerstiens.com/2013/01/10/more-on-postgres-performance/


18



Tools• pbBadger: dalibo.github.io/pgbadger/

• Requires Perl

pgBadger: dalibo.github.io/pgbadger/postgresql.conf• log_min_duration_statement = 0• log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d ’• log_checkpoints = on• log_connections = on• log_disconnections = on• log_lock_waits = on• log_temp_files = 0After making changes run: SELECT pg_reload_conf();

http://dalibo.github.io/pgbadger/







19


Implement and test (…and repeat...)

Supporting points• What if I don’t find the offending SQL or table? • Start looking at the database configuration.• Did you benchmark and tune?

More points• Most parameters are automatically adjusted.• We find that many people make minimal changes to the default

configuration. • As always, remember to test any configuration changes in a non-

prod environment first, and implement changes incrementally. • Do not make numerous configuration changes all at once. • Systematically measure your performance tests. Use a tool like

pgbench.


20



Configuration Settings in postgresql.conf• max_connections: Maximum simultaneous connections to

the database (default is 100).• shared_buffers: database cache size (default is 128MB).

25% of total RAM. Windows-based should be 64-512MB.• effective_cache_size: Tells the query planner how much

RAM there is to execute. 50-75% of total RAM.• work_mem: Used for complex sorts (default is 4MB). Check if

it is uncommented and what the value is.• maintenance_work_mem: Amount of memory for background

processes for pgdump, pgrestore, vacuum, indexing, and bulk loads (default is 64MB). 256MB-1GB for large databases.

• checkpoint_segments: Maxiumum # of log file segments between WAL checkpoints (default is 3).


21



Configuration Settings in postgresql.conf• wal_buffers: Write ahead log buffer used for writing a

transaction to disk.

What should I really set these values to?• pgTune: pgtune.leopard.in.ua/

SELECT name, current_setting(name), SOURCE FROM pg_settings WHERE SOURCE NOT IN ('default', 'override'); PostgreSQL 9.0 High Performancewww.amazon.com/PostgreSQL-High-Performance-Gregory-Smith/dp/184951030X/163-3733534-8577963

http://pgtune.leopard.in.ua/

http://www.amazon.com/PostgreSQL-High-Performance-Gregory-Smith/dp/184951030X/163-3733534-8577963

http://www.amazon.com/PostgreSQL-High-Performance-Gregory-Smith/dp/184951030X/163-3733534-8577963


22

Poll #4Are you using SSLv3?

A: YesB: No


23

Top issues:security


24

Scenario #3

Pen testing reveals the application server is vulnerable

Supporting points• Cache-control, Pragma, Expires headers are not

set• X-Powered-By reveals the application server

typeMore points• A proxy can inject an intermediate

page and compromise the site• Certain versions of JBoss allow

little to no control of request header manipulation


25


Request header manipulation strategiesSupporting points• Proxy requests through Apache HTTP Server before

sending them to the application server• Mod_expire could be used to explicitly set them. Does

not guarantee header will be set• Set org.apache.catalina.connector.X_POWERED_BY to

falseMore points• Using Apache HTTP Server’s mod_header module is easiest and

preferred option• mod_headers module allows manipulation before and after the

request• If this is not an option, then a filter can do the same function• There should never be a situation where an application server is

internet facing without the protection of a web server in the DMZ


26


Request header manipulation strategiesDMZ• The front end should always be in the DMZ• This should never be an application server • Web servers are “far more” capable than

application servers in this role

Best practice• Traditional 3-headed monster, Web/App/DB should always

reside at unique layers in the infrastructure• There should be horizontal and/or vertical separation between

them at all times• This provides greater opportunities for other security gear to

potentially sniff and sort out other possible vulnerabilities. Like HIDS/NIDS, etc.


27


Request header manipulation strategiesHTTPD Mod_expires• Mod_expires can set the max_age and

expirations• ExpiresByType text/html M604800 (Expires a

week after modification)HTTPD Mod_Header• Much more flexible than mod_expires• Header echo ^KC (copies all request headers starting with KS

to response headers• Header set TESTHEADER “Hi, The request ran in %D uSEC’s”• Env variables can be set as well. Much more capable than

mod_expires


28


Header manipulation examplesHTTPD Mod_header Example• <ifModule mod)headers.c>• Header set DateTimeHEader “%D %t”• </ifModule>

Header merge Cache-Control no-cacheHeader merge Cache-Control no-store

YieldsCache-Control: no-cache, no-store


29


Header manipulation examplesHTTPD Mod_header Example• Header set Set-Cookie testcookie “expr=-z %

{req:Cookie}”• Header merge Cache-Control no-cache env=CGI• Header merge Cache-Control no cache

end=NO_CACHE• Head merge Cache-Control not-store

env=NO_STORE

Non-exclusive conditions. If all are set: Cache-Control: no-cache, no-store

Using append instead of merge would cause a duplicate no-cache message


30


Header manipulation examplesHTTPD Mod_header (Powerful module)• Add, Append, echo, edit, merge, set setifempty,

unset• Allows setting of environment variables to use

are triggers• Expressions can be set as well for more

additional control

Examples prove that web servers are well suited for this type of work.


31


Request header manipulation strategiesJBoss Wildfly : Header Manipulation• You shouldn’t, but if you absolutely must• You could modify system properties to override

certain values. Or set a filter.• Newer versions of JBoss provide header

modificationExample• <system properties>• <property

name=“org.apache.coyote.http11.Http11Protocol.SERVER” value=“someserver”/>

• <system-properties>• For each version of Jboss, you may need to use the CLI to

determine if the necessary keys are available in that version


32


Request header manipulation strategies

JBoss Wildfly : Header Manipulation• You can, but apache httpd is still a good accomplice• Remove the following the alter the headers• <filter-ref name=“x-powered-by-header”/>• Performed in the undertow segment of

standalone.xml

Example• As newer versions of Wildfly emerge there is a bit more control

of headers• This does not remove the need to have a web server in front• A reverse proxy is is vital piece of application server security


33


Request header manipulation strategies

JBoss Wildfly : Header Manipulation• CLI command structure will provide a view of current

header manipulation capabilities Wildfly has• More flexibility in each new release• Wildfly’s capabilities are far behind that of apache’s• Having this level of control at your reverse proxy just

makes the most sense• These features have been requested for quite some

time from the user base.• Not a replacement for a frontend DMZ based web

server


34

Who wrote LevelDB and what version of ActiveMQ did it debut in?Written by Google: Jeff Dean who also has contributed to MapReduce and Google TranslateSanjay Ghemawat who also contributed to MapReduce and iCal which is a popular calendar application in Unix/Linux.

LevelDB debuted in ActiveMQ 5.10.0 but was not production ready.What percentage of web server market share does Apache HTTP Server hold?July 2016Apache HTTP Server: 52%nginx: 30.5%Microsoft: 12%

Pop quiz answers

https://w3techs.com/technologies/overview/web_server/all


35

Conclusion


36

JBoss/Wildfly: Newer versions of Wildfly allow administrators to control HTTP headers and older versions of JBoss can be augmented with a filter or Apache HTTP Server's mod_header for equivalent functionality.

So much open source!

PostgreSQL: Use tools like pgBench and pgTune.

PostgreSQL: Troubleshooting tools like pg_stat_statements and pgBadger can help narrow down database performance issues.

ActiveMQ: A Lease Database Locker is a viable alternative to NFS where SAN is cost prohibitive. Consider Replicated LevelDB with Apache Zookeeper for performance.


37

Our support


38

Q & A


39

Watch on demand

• Watch this webinar on demand

• Read the recap blog to see the results of the polls and Q&A session

http://www.roguewave.com/events/on-demand-webinars/top-issues-in-the-top-enterprise-packages

http://blog.klocwork.com/open-source/top-issues-in-the-top-enterprise-packages-activemq-postgresql-jbosswildfly/


40

Follow up

For OpenLogic support customers:

OSS Radio

Get a free OSS support ticket to experience our expertise

roguewave.com/freeticket

Free open source newsletter:

roguewave.com/products/open-source-support/openupdate

http://www.roguewave.com/freeticket

http://www.roguewave.com/freeticket

http://www.roguewave.com/products/open-source-support/openupdate

http://www.roguewave.com/products/open-source-support/openupdate


41

open source: top issues in the top enterprise packages

Technology