deploying distributed databases and in-memory …...© 2018 gridgain systems, inc. gridgain company...

©2018GridGainSystems,Inc. GridGainCompanyConfidential

Deploying Distributed Databases andIn-Memory Computing Platforms with Kubernetes

Denis MagdaGridGain Product ManagementApache Ignite PMC Chair


Agenda

• DeploymentSpecificitieswithKubernetes• Memory-OnlyDeployments• StatefulDeployments• ManagementandMonitoring• Demo• Q&A


DeploymentSpecificitieswithKubernetes


DistributedDatabaseSpecificities

• Databaseisasetofpods– IPsareassigneddynamically– Auto-discoveryisneeded

• ApplicationsDeployment– WithinKubernetes?– NotmanagedbyKubernetes

• StatelessorStateful?


DatabasePodsAuto-Discovery

• KubernetesLookupService– TracksalistofallIgnitepods– Gatewayforremoteapps

• KubernetesIPFinder– ConsumesIPsfromtheservice– Let’snodetojointhecluster


ServiceStartupapiVersion: v1kind: Servicemetadata:

name: ignitenamespace: ignite

spec:type: LoadBalancerports:

- name: restport: 8080targetPort: 8080

- name: sqlport: 10800targetPort: 10800

- name: thinclientsport: 10900targetPort: 10900

selector:app: ignite

kubectl create -f ignite-service.yaml

IPFinderConfiguration

<bean class="org.apache.ignite.configuration.IgniteConfiguration"><property name="discoverySpi">

<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

<property name="ipFinder"><bean class=”

org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder"> <property name="namespace" value="ignite"/>

</bean></property>

</bean></property>

</bean>


Memory-OnlyDeployments


Memory-Centric StorageScale to 1000s of Nodes & Store TBs of Data

Ignite Native Persistence(Flash, SSD, Intel 3D XPoint)

Third-Party PersistenceKeep Your Own DB

(RDBMS, HDFS, NoSQL)

SQL Transactions Compute Services MLStreamingKey/Value

IoTFinancialServices

Pharma &Healthcare

E-CommerceTravel & LogisticsTelco

ApacheIgniteOverview


IgniteMemoryUsageModes

Mode Description Major Advantage

In-Memory Pure In-Memory Storage Maximum perfomance possible(data is never written to disk)

In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc

Horizontal scalabilityFaster reads and writes

In-Memory + Full Copy on Disk The whole data set is stored both in memory and on disk Survives cluster failures

100% on Disk + In-Memory Cache 100% of data is in Ignite native persistence anda subset is in memory

Unlimited data scale beyond RAM capacity


IgniteasKubernetesDeploymentEntityapiVersion: extensions/v1beta1kind: Deploymentmetadata:

name: ignite-clusterspec:

replicas: 2template:

metadata:labels:

app: ignitespec:

containers:- name: ignite-node

image: apacheignite/ignite:2.5.0env:- name: OPTION_LIBS

value: ignite-kubernetes- name: CONFIG_URI

value: URL_TO_CONFIGports:...

kubectl create -f ignite-deployment.yaml


StatefulDeployments


StatefulDeployments

• DurabilityWithStatefulSet– Datapersistencetodisk– Orderedrestarts

• SeparateVolumesfor– Dataandindexes– WAL(akaTransactionLog)– Snapshotsandbackups

apiVersion: apps/v1beta2kind: StatefulSetmetadata:

name: ignitenamespace: ignite

spec:selector:

matchLabels:app: ignite

serviceName: ignite...

volumeMounts:- mountPath: "/data/ignite"

name: ignite-storagevolumeClaimTemplates:- metadata:

name: ignite-storagespec:

accessModes: [ "ReadWriteOnce" ]resources:

requests:storage: 1Gi


ClusterActivation

• ManualActivationonFirstStart

kubectl exec -it ignite-0 --namespace=ignite -- /bin/bashcd /opt/ignite/apache-ignite-fabric/bin/./control.sh --activate

• AutomaticActivationonRestarts– Baselinetopologyusage


ManagementandMonitoring


K8DashboardandIgniteWebConsole

• KubernetesDashboard– ForKubernetesenvironment

• IgniteWebConsole– ForIgnitecluster– DeployWebAgentinK8


Demo


ApacheIgnite– We’reHiring;)

• VeryActiveCommunity• GreatWaytoLearnDistributedComputing• HowToContribute:

– https://ignite.apache.org/


AmongTop5ApacheProjects

Top5byCommits

1. Hadoop2. Ambari3. Camel4. Ignite5. Beam

Top5DeveloperMailingLists

1. Ignite2. Kafka3. Tomcat4. Beam5. James

Over1Mdownloadsperyear

Top5UserMailingLists

1. Lucene/Solr2. Ignite3. Flink4. Kafka5. Cassandra


Thank you for joining us. Follow the conversation.https://ignite.apache.org

https://gridgain.com/

Q&A

@denismagda#apacheignite#gridgain

deploying distributed databases and in-memory …...© 2018 gridgain systems, inc. gridgain company...

Documents