solarcloud – rapid data loggingfrom a solar power...

SolarCloud – Rapid Data Logging| 1

SolarCloud – Rapid Data Loggingfrom a Solar Power Plant

AUTHOR: TEODOR TALOV

COP 4908 – Independent Study Instructor: Janusz Zalewski

Florida Gulf Coast University Fort Myers, FL

July25, 2013

Final Draft


1. Introduction

The objective of this project is to provide an overview of a non‐relational database using

MongoDB as an example, compare its performance with MySQL, and test its operations for data

storage from the solar panel node, as a case study.

The rest of the document is structured as follows: Section 2 describes the principles of storing

data in MongoDB; Section 3 presents the comparison methodology, and Section 4 describes a

case study of the Solar Plant. Section 5 describes some comparison in performance between

MongoDB and MySQL. Both databases are compared with different length of records as well as

different number of records 10, 100, 1000, 10 000 respectively. Finally, Section 6 outlinessome

conclusions.


2. MongoDB

2.1. Overview of MongoDB

MongoDB is a shared‐nothing, document‐oriented database management system. Unlike

relational database systems, which store all data in tables defined by a schema, MongoDB

stores data in schema‐less JSON documents (Fig 1). Users perform queries against the

database using procedural API calls, rather than using a declarative language like SQL. It

combines the ability to scale out with many of the post powerful features of relational

databases, such as secondary indexes, range queries, and sorting. MongoDB is also

incredibly featureful: it has tons of useful features such as built‐in support for MapReduce

style aggregation and geospatial indexes.

Fig 1. Example of MongoDB document structure

Mongo is a document‐oriented database, not a relational one. The primary reason for

moving away from the relational model is to make scaling out easier, but there are some

other advantages as well. In a relational database scaling out becomes challenging once the

cluster reaches capacity and adding more storage becomes necessary. Additional

complexity is introduced by adding more servers because in a relational database is not

able to determine how to interact with multiple servers. Most common scenario is master‐

slave replication, which works very well for applications relaying heavily on Read

operations, but it is not very effective for Write heavy applications, simply because there


must be only one node, the master node, containing the complete database, which gets

copied over to all slave nodes.

In contrast, MongoDB, and most NoSQL databases for that matter, come with built‐in

scalability. Specifically, once capacity is reached, more servers can be added to the cluster,

but MongoDB will handle how they are used automatically. It can balance data and load

across a cluster, redistributing documents automatically. When they need more capacity,

they can just add new machines to the cluster and let the database figure out how to

organize everything.

The basic idea is to replace the concept of a “row” with more flexible model, the

“document”. By allowing embedded documents and arrays, the document‐oriented

approach makes it possible to represent complex hierarchical relationships with a single

record. This fits very naturally into the way developers in modern object‐oriented

languages think about their data.

MongoDB is also schema‐free: a document’s keys are not predefined or fixed in any way.

Being schema‐free does not mean that there is no schema to the database. To the contrary,

there is schema to the database, but it is not strictly enforced by MongoDB. Without a

schema to change, massive data migrations are usually unnecessary. New or missing keys

can be dealt with at the application level, instead of forcing all data to have the same

shape. This gives developers a lot of flexibility in how they work with evolving data models.

MongoDB was designed from the beginning to scale out. [4]

MongoDB also provides most if not all features that a relational databases provides such as

indexing, stored procedures, aggregation, etc. Specifically, indexing supported by MongoDB

is generic secondary indexes, allowing a variety of fast queries and it provides unique,

compound, and geospatial indexing capabilities as well. Furthermore, the stored


procedures that MongoDB supports are simply JavaScript functions. For aggregation

MongoDB supports MapReduce amongst many other features such as aggregate,

group, etc.

Concerning security MongoDB has provided multiple mechanisms such as, Access Control

and Network Security. Furthermore, 10gen [3] provides a security guide which takes

a Defense in Depth approach to securing MongoDB deployments and addresses a number

of different methods for managing risk and reducing risk exposure.The intent of a Defense

In Depth approach is to ensure there are no exploitable points of failure in the deployment

that could allow an intruder or untrusted party to access the data stored in the MongoDB

database. The easiest and most effective way to reduce the risk of exploitation is to run

MongoDB in a trusted environment, limit access, follow a system of least privilege, and

follow best development and deployment practices.

2.2. Installation Guide to MongoDB [3]

To install MongoDB the following steps have to be followed:

2.2.1. Add 10gen package to source.list.d by running the following commands:

sudo apt-cache search mongodb

mongodb

mongodb-clients

mongodb-dev

mongodb-server

2.2.2. Add the following line to /etc/apt/sources.list.d/mongo.list:

deb http://downloads-distro.mongodb.org/repo/ubuntu-

upstart dist 10gen


Note: If the file does not exist, create it.

2.2.3. Add GPG Key in order to verify the authenticity of the MongoDB repository by running

the following command:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv

7F0CEB10

2.2.4. Update Ubuntu by running the following command:

sudo apt-get update

2.2.5. Install MongoDB by running the following command:

sudo apt-get install mongodb-10gen

2.2.6. Verify that MongoDB is running by executing the following command:

mongo

If MongoDB is running, the screen is shown on Fig. 2

The a

system

69.8

user

passw

Note:

Fig. 2

pplication is h

m. The server

8.163.13 o

name: keyw

word: same

keywest u

2 MongoDB S

hosted on the

r can be acces

or http://

west

e as VMWARE

ser has admi

Shell

e Key West se

ssed using th

keywest.cs

E host pas

nistrative priv

erver, which h

e following c

s.fgcu.edu

ssword

vileges (sudo

Sola

has Ubuntu 1

redentials:

/

er)

arCloud – Rap

12 installed as

pid Data Loggi

s operating

ing| 7


3. MongoDB vs. MySQL comparison methodology

General concepts have to be evaluated in both MongoDB and MySQL. Additionally, features in

MongoDB need to be compared to their respective features in MySQL and then their

performance evaluated. Performance is evaluated by running heavy database queries, 100 000,

500 000, 1 00 000 records respectively, for each of the features outlined below. Finally,

operators and syntax such as > and $gthave to be compared in both MySQL and MongoDB

3.1. General Concepts

3.1.1. Database

Database is an organized collection of data. Both MySQL and MongoDB use databases

to store data.

MySQL – MySQL is an open source relational database management system.

Information in a MySQL database is stored in the form of related tables. MySQL does

have multiple storage engines such as MyISAM and InnoDB.

MongoDB ‐ A physical container for collections. Each database gets its own set of files

on the file system. A single MongoDB server typically servers multiple databases.

MongoDB does not support different storage engines.

3.1.2. Table vs. Collection

MySQL – Since MySQL is relational database management system (RDMS) all the data is

stored in tables, which are organized according to the relation model. The relational

model for database management is a database model based on first‐order predicate

logic, first formulated and proposed in 1969 by Edgar F. Codd. In the relational model of


a database, all data is represented in terms of tuples, grouped into relations. A database

organized in terms of the relational model is a relational database.

MongoDB – Unlike MySQL MongoDB does not use tables, but it uses COLLECTIONS.

Collections are groupings of BSON documents (A serialization format used to store

documents and make remote procedure calls in MongoDB. “BSON” is a portmanteau of

the words “binary” and “JSON”. Think of BSON as a binary representation of JSON

(JavaScript Object Notation) documents). Collections do not enforce a schema, but they

are otherwise mostly analogous to RDBMS tables. The documents within a collection

may not need the exact same set of fields, but typically all documents in a collection

have a similar or related purpose for an application. All collections exist within a single

database. The namespace within a database for collections are flat.

3.1.3. Row vs. Document

MySQL – Since MySQL is a relational databases, it uses rows. In the context of a

relational database, a row—also called a record or tuple—represents a single,

implicitly structured data item in a table. In simple terms, a database table can be

thought of as consisting of rows and columns or fields. Each row in a table represents a

set of related data, and every row in the table has the same structure. For example, in

a table that represents companies, each row would represent a single company.

Columns might represent things like company name, company street address, whether

the company is publicly held, its VAT number, etc.. In a table that represents the

association of employees with departments, each row would associate one employee

with one department. [5]


MongoDB stores structured data as JSON‐like documents, using dynamic schemas

(called BSON), rather than predefined schemas. In MongoDB, an element of data is

called a document, and documents are stored in collections. One collection may have

any number of documents. [3]

3.1.4. Column vs. Field

MySQL – MySQL uses columns. In the context of a relational database table, a column is

a set of data values of a particular simple type, one for each row of the table. The

columns provide the structure according to which the rows are composed. [5]

MongoDB – On the other hand MongoDB uses fields. A name‐value pair in a document.

Documents have zero or more fields. Fields are analogous to columns in relational

databases. [3]

3.1.5. Indexes

Index is a data structure that improves the speed of data retrieval operations on

a database table at the cost of slower writes and the use of more storage space.

Indexes can be created using one or more columns of a database table, providing the

basis for both rapid random lookups and efficient access of ordered records.

Both MySQL and MongoDB index data in the same way.

3.1.6. Table joins vs. embedded documents and linking

Traditional SQL databases use table joins. A table join combines records from two or

more tables in a database. It creates a set that can be saved as a table or used as it is.

A JOIN is a means for combining fields from two tables by using values common to

each.MySQL resolves all joins using a single‐sweep multi‐join method. This means that

MySQL reads a row from the first table, and then finds a matching row in the second


table, the third table, and so on. When all tables are processed, MySQL outputs the

selected columns and backtracks through the table list until a table is found for which

there are more matching rows. The next row is read from this table and the process

continues with the next table.MongoDB does not support joins. In MongoDB some

data is denormalized, or stored with related data in documents to remove the need for

joins. However, in some cases it makes sense to store related information in separate

documents, typically in different collections or databases.

3.1.7. Primary key

The PRIMARY KEY constraint uniquely identifies each record in a database table.A

record’s unique, immutable identifier. In anMySQL, the primary key is typically an

integer stored in each row’s id field. In MongoDB, the _id field holds a document’s

primary key which is usually a BSON ObjectId.

3.1.8. Aggregation

Aggregate functions perform a calculation on a set of values and return a single value.

The data that needed is not always stored in the tables. However, it can obtain by

performing the calculations of the stored data when you select it. Except for COUNT,

aggregate functions ignore null values. Aggregate functions are frequently used with

the GROUP BY clause of the SELECT statement. Both MongoDB and MySQL use

aggregation similarly. However, MongoDB performs aggregation via its aggregation

framework.

3.2. Programming Features

This section is a side‐by‐side comparison of MySQL versus MongoDB. It compares their


functions such as Select vs. Find, Update, Delete, and other operations. It also provides

syntax comparison for both database types – MySQL and MongoDB.

3.2.1. Select vs. Find operations

MySQL the SQL select statement returns a result set of records from one or more tables

[5]. A select query looks like:

SELECT * FROM users

MongoDB uses find in order to retrieve data sets by given criteria. A find statement looks

like:

db.users.find()

3.2.2. Insert operations

MySQL uses its insert statement in order to add rows to a table. An insert statement as

follows:

INSERT INTO users(user_id, age, status) VALUES ("bcd001", 45,

"A")

MongoDB uses its insert statement in order to insert a record to a document. An insert

statement looks like this:

db.users.insert( { user_id: "bcd001", age: 45, status: "A”}

)

3.2.3. Update operations


MySQL uses its update statement in order to update existing records in a table. An

update statement looks like this:

UPDATE users SET status = "C" WHERE age > 25

MongoDB uses its update statement in order to update existing records in a document.

An update statement looks like this:

db.users.update({ age: { $gt: 25 }},{$set:{ status: "C"

}},{ multi: true })

3.2.4. Delete operations

MySQL uses its delete statement in order to delete existing records from a table. An delete

statement looks like this:

DELETE FROM users WHERE status = "D"

MongoDB uses its delete statement in order to delete existing records from a document.

An delete statement looks like this:

db.users.remove( { status: "D" } )

3.3. Operators

3.3.1. Logical Operator AND

MySQL logical AND operator compares two expressions and returns true if both of the

expressions are true.

In MongoDB a logical AND joins two clauses and returns all documents that match the

conditions of two clauses.

3.3.2. Logical Operator OR

MySQL OR operator compares two expressions and returns TRUE if either of the


expressions is TRUE.

In MongoDB a logical ORoperator, joins query clauses with all documents that match

the conditions of either clause.

3.3.3. Logical Operator NOT

MySQL NOT operator reverses or negates the input.

In MongoDB NOT inverts the effect of a query expression and returns documents that

do notmatch the query expression

3.4. Essential Functions

3.4.1. Order By vs. Sort

In MySQL the SQL ORDER BY clause allows sorting records in a result set. The SQL

ORDER BY clause can only be used in SQL SELECT statements.

In MongoDB sort() is returns a cursor with documents sorted according to a sort specification.

3.4.2. Where

In MySQL SQL WHERE clause allows you to filter the results from an SQL statement ‐

SQL SELECT statement, SQL INSERT statement, SQL UPDATE statement, or

SQL DELETE statement.

In MongoDB $where returns that match documents that satisfy a JavaScript

expression.

3.4.3. Max

In MySQL MAXfunction returns the maximum value of an expression.

In MongoDB max() specifies a minimum exclusive upper limit for the index to use in a

query


3.4.4. Min

In MySQL MINfunction returns the maximum value of an expression.

In MongoDB min() specifies a minimum exclusive upper limit for the index to use in a

query


4. Case Study

4.1. Overview

FGCU’s Solar Plant (Fig. 3) takes up 760 acres of its campus, which makes it the second

larges solar installation located in a University campus in the U.S. [1]. The plant began its

operations in 2009 and it is expected that the 2‐megawatt Solar Plant will generate 15% of

the electricity that the University consumes [2].

Fig 3: Solar Plant Eagle View

Additionally, the Solar Plant includes a separate small site where different panels can be tested

and demonstrated (Fig. 4).

Fi

Th

w

in

4

Th

is

it

ad

In

cu

fr

se

ig 4. Experime

he smaller ins

with the main

ndeed, the sit

.2 Problem D

he power cur

s mandated b

s mandate, th

dditional hard

n order to solv

urrently supp

rom the Solar

econd. Theref

ental Solar Pl

stallation is v

solar plant, b

te where all u

escription

rrent generat

y the State of

he Solar Plant

dware (back‐

ve that probl

ported by the

r Plant every 1

fore, a softwa

ant at FGCU

ery safe and

but provides t

pcoming exp

ed by the Sol

f Florida that

t cannot cont

up generator

em, data mus

existing syste

1 minute. How

are applicatio

convenient fo

the same exa

eriments can

ar Plant is no

power curre

tribute power

rs), which will

st be analyze

em. The curre

wever, the de

on has to be d

SolarC

or experimen

ct configurat

n be conducte

ot steady, thu

nt must be st

r to the powe

l kick‐in when

d at much hig

ent system ha

esired freque

developed tha

Cloud – Rapid

nts, since it do

ion as the ma

ed [6].

s energy fluct

teady at all tim

er grid withou

never the cur

gher frequenc

as the ability

ency for data a

at will acquire

d Data Loggin

oes not interf

ain plant. This

tuations occu

mes. Because

ut installing

rent is not ste

cy than it is

to acquire da

analyses is ev

e data with th

ng| 17

fere

s is,

ur. It

e of

eady.

ata

very 1

he


desired frequency.

Fig. 5Solar Plant data acquisition diagram

The primary objective of the project is to acquire data from the experimental solar plant in 1

second intervals and record these data into a database. Furthermore, the data have to available

on the Internet and accessible upon successful authentication.

The proposed system outlined in Fig. 5, acquires data every second.The SolarCloud

softwareresiding on the webserver sends a request to the CR 1000 data logger via its Ethernet

Interface NL 120.As a response the CR 1000 sends back a JSON or XML object containing a data

structure with measurement results, which is stored in a database by SolarCloud software. The

end user (public client) will be able to generate reports based on the data that are already

stored in the database.

4.3Project Objective and Technology

4.3.1. Data Logger

To

sy

6)

Th

te

p

an

ac

co

P

an

In

In

10

10

o acquire dat

ystem. For da

) will be used

he CR1000's m

elecommunic

rograms in on

nd glitch prot

ccurate timek

ommunicatio

rogramming

nd receive a r

n order to allo

nternet conne

0baseT Ether

0baseT Ether

ta from the so

ata acquisition

d [7].

Fig 6:

module acqu

cations, reduc

n‐board, non‐

tected by the

keeping. The

on functions[7

Interface (AP

response in a

ow the CR100

ction via TCP

rnet straight t

rnet crossove

olar plant eve

n from the So

Campbell Sci

ires measure

ces amount of

‐volatile stora

sealed, stain

module can s

7]. Campbell S

PI) which allow

n XML or JSO

00 data logge

P/IP a NL 120

hrough cable

er cable is use

ery second as

olar Plant a Ca

ientific CR 10

ment from se

f data, contro

age. The elect

nless‐steel can

simultaneous

Scientific CR 1

ws third party

ON format.

r to communi

0 Ethernet (Fig

e when the ca

ed if the cable

SolarC

opposed to e

ampbell Scien

00 Data logge

ensors, drives

ols external d

tronics are sh

nister. A batte

sly provide me

1000 Data log

y applications

icate over a lo

g. 7) interface

ble is run from

e is run direct

Cloud – Rapid

every minute

ntific CR 1000

er

s direct comm

evices, and st

hielded from

ery‐backed cl

easurement a

gger has a bu

s to authentic

ocal network

e is used [6]. T

m a hub to the

tly from the co

d Data Loggin

as in current

0 Data logger

munications a

tores data an

RF interferen

lock assures

and

ilt‐in Applicat

cate, request

or a dedicate

The NL120 u

e NL120. A

omputer to the

ng| 19

t

(Fig.

nd

nd

nce

tion

data,

ed

ses a

e

N

28

4

NL120. For ca

8033 surge p

.3.2 Runnin

Usage of a

currently i

Ch

Ch

Ch

Ch

Ch

ble lengths lo

protector need

ng the App (F

available chan

n use as follo

hannel 1: Syst

hannel 2: Shu

hannel 3: Shu

hannel 4: Pan

hannel 5: Tilt

onger than 9 f

ds to be used

Fig 7: NL 12

Fig. 13).

nnels of CR‐10

ows:

tem Volts

nt Volts

nt Tem Senso

el Temperatu

Sensor

feet, either a s

[6].

20 Ethernet in

000 data logg

or

ure Sensor

SolarC

shielded 10ba

nterface

ger are shown

Cloud – Rapid

aseT Etherne

n on Fig 8. The

d Data Loggin

et cable or the

ere are 5 cha

ng| 20

e

nnels

Fi

4

To

us

ig. 8 Proposed

.3.3 Configura

o interface w

sed.

d use of data

ation of CR‐1

with the CR‐10

channels

000 data logg

000 Campbell

ger.

Scientific’s P

SolarC

PC200W supp

Cloud – Rapid

lied software

d Data Loggin

e (Fig. 9) must

ng| 21

t be


Fig. 9 ‐ PC200W user interface

The data logger must be connected to a Microsoft Windows PC via serial cable via its serial port

RS‐232 (shown in Fig 10).

Fig 10 ‐ Port RS‐232 of the data logger.

After the data logger is successfully connected with the cable and the software to the PC, the

program found in Appendix A must be uploaded to the data logger. This will reconfigure the

data logger with the appropriate tables. In this Case Study the parameters are shown in Table 1.

Each row corresponds to a channel of the Data Logger.

Voltage1 Voltage2 Voltage2 Voltage4 Voltage5

Table 1. Table used by the Data Logger in this Case Study.

Additionally, a static IP must be assigned to the data logger in order for other devices to be able

to request data from it. This is done via the Device Configuration Utility within the

PC200Wprogram as shown on Fig 11 below.

Fi

A

4

Th

se

ig 11. Device

ssign the follo

Bind A

(more

.3.2 Database

he software a

econd interva

Configuration

owing measu

Average Volta

to come here

e platform ‐ S

application ha

als and record

n Utility by Ca

urements to e

ge to Channe

e once we ins

SolarCloud

as been deve

ds the reading

ampbell Scien

each channel:

el 1

stall the data

loped which

gs in a databa

SolarC

ntific

logger)

acquires the

ase. Finally, th

Cloud – Rapid

data from the

he data have

d Data Loggin

e Solar Plant

to be presen

ng| 23

at 1

ted


to the end user. The following technology stack shown in Fig 12 is suitable for use in this case:

Fig 12. Technology stack

4.3.3.1 Installation of the Server Environment

The following technologies have to be installed on the server in this order (admin privileges

required)

Operating System: Ubuntu 12.x [6]

Installation notes: Isntall Ubuntu using a LiveCD with the following network

configuration:

auto eth0

face eth0 inet static

address 69.88.163.13

netmask 255.255.255.0

network 69.88.163.0

broadcast 69.88.163.255

gateway 69.88.163.1

dnssearch cs.fgcu.edu

dns-nameservers 172.28.254.2 172.28.254.3

Solar Cloud Application

MongoDB

NodeJS

Linux


Web Server: Apache 2.2x [6]

Run the following commands to install Apache

sudo apt-get install tasksel

sudotasksel install lamp-server

Database: MongoDB [6]

See section 2 for installation notes:

Back‐end technologies: NodeJS [6]

Run the following commands in order to install NodeJS

sudo apt-get install python-software-properties python g++

make

sudo add-apt-repository ppa:chris-lea/node.js

sudo apt-get update

sudo apt-get install nodejs

Version control: Git

Installation notes: Run the following command in order to install Git

sudo apt-get install git

4.3.3.2 SolarCloud Installation

Solar Cloud software has been developed in the Spring 2013 project [6]. The general design is

sh

Ex

hown in Fig 1

xecute the fo

Create

sudom

Naviga

cd /e

Clone

acqui

sudog

nodej

HTTPS

git c

nodej

3.

ollowing steps

e a solarcolud

mkdir /et

ate to the Sol

etc/www/s

the GitHubre

isition-n

git clone

js-mongod

S by running t

clone htt

js-mongod

s in order to i

d directory un

tc/www/sol

arCloud direc

olarcloud

epositorygit

nodejs-mon

e git@gith

db.git . .

he following

tps://gith

db.git .

nstall SolarCl

nder /etc/w

larcloud

ctory by runn

d

[email protected]

ngodb.gitb

hub.com:te

. If port 22 is

command:

hub.com/te

SolarC

oud

www by runnin

ing the follow

com:teodor

by running th

eodortalo

s blocked the

eodortalo

Cloud – Rapid

ng the followi

wing comman

rtalov/da

he following c

ov/data-ac

e repostiroy c

ov/data-ac

d Data Loggin

ing command

nd

ata-

command

cquisition

ould be clone

cquisition

ng| 26

d

n-

ed via

n-

4.4 T

T

b

D

s

Install

sudon

4.3.3.3 Sol

In

cd

no

In

cd

no

Troubleshoot

There are mu

below as well

Data logger d

shown on Fig

Express Fram

npm insta

larCloud Start

order to star

d /etc/www

ode fetche

order to star

d /etc/www

ode app(sho

ing

ltiple known

as suggested

oes not respo

14

Fig 1

mework by ru

all

t

rt the Fetcher

w/solarcl

er

rt the App, ru

w/solarcl

own on Fig 12

issues with th

d resolutions

ond with the

14. Incorrect r

nning the fol

r run the follo

loud

n the followin

loud

2)

he current co

for each of th

expected JSO

response from

SolarC

lowing comm

owing comma

ng command

onfiguration. A

hem

ON data struct

m data logger

Cloud – Rapid

mand:

ands:

s:

All known iss

ture. Example

r

d Data Loggin

ues are listed

e of the issue

ng| 27

d

e is


This issue might be a result of the internal memory of the data logger being full. Suggested resolution is

reconfiguring the data logger to its initial state using the program included in Appendix A.

If when connecting to MongoDB the following error is thrown: JavaScript execution failed: Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:L112 exception: connect failed run the following command to resolve the issue:

sudorm /var/lib/mongodb/mongod.lock


5. Experiements

Experiments are ran on two separate servers, a development server and production server (Key

West server). Both have Ubuntu 12.4 with PHP 5.4, MySQL 5.5 and MongoDB 2.4.

Some experiementation was conducted with READ and WRITE operations for both MySQL and

MongoDB. Also experimentation was done with the following number of records: 10, 100, 1 000,

10 000. Each experiment is ran with two different record lenghts, 10 characters and 100

characters. Diagram of server infrastructure for both MongoDB and MySQL is shown on Fig 15

Fig 15 – Server Infrastructure for MySQL and MongoDB

PHP was used as a mid‐ware and InnoDB was used as MySQL’s storage engine. The experiments

were run on two servers with indentical configuration but different server resources.

Server 1 ‐ Processor 2.4GHz and RAM: 16.00GB

Server 2 (Key West) – Processor 1.98GHz and RAM: 1.96GB

The resutls from WRITE are shown on below:

Ubuntu

Apache webserver

PHP

MySQL

Ubuntu

Apache webserver

PHP

MongoDB


Server 1

10 characters record lenght:

WRITE operations

Number of records MongoDB execution time (sec) MySQL execution time (sec)

10 0.001223087310791 0.10456109046936

100 0.0016720294952393 0.021401882171631

1000 0.026886940002441 1.5926580429077

10 000 0.26401996612549 14.037951946259

Table 2 – Timing WRITE operations of Server 1 with 10 characters record lenght

READ operations

Number of records MongoDB execution time (sec) MySQL execution time(sec)

10 0.01720830917365 0.011533975601196

100 0.021724939346313 0.014551162719727

1000 0.029290790557864 0.014918804168701

10 000 0.021161079406738 0.0207

Table 3 – Timing of READ operationsof Server 2 with 10 characters record lenght



WRITE operations


10 0.0012331008911133 0.051568031311035

100 0.011001110076904 0.22046899795532

1000 0.033560991287231 1.2954359054565

10 000 0.3572781085968 17.920379161835


READ operations


10 0.051701784133911 0.0072619915008

100 0.0712956509590149 0.0093290596008301

1000 0.099290790557864 0.0107019462585449

10 000 0.92059993743923 0.092409925460815

Table 5 – Timing of READ operations of Server 2 with 100 characters record lenght


Server 2


WRITE operations


10 0.0024571418762207 0.011458873748779

100 0.0044529438018799 0.10197496414185

1000 0.042062997817993 0.64150214195251

10 000 0.51646995544434 6.0016031265259


READ operations


10 0.012411832809448 0.045092105865479

100 0.10247802734375 0.035649061203003

1000 0.64187788963318 0.096686916351318

10 000 1.0019898414612 0.143524961471558




WRITE operations


10 0.0021369457244873 0.0090398788452148

100 0.0076048374176025 0.059972047805786

1000 0.047162055969238 0.62525701522827

10 000 0.050267934799194 6.0794649124146


READ operations


10 0.0099599361419678 0.036328077316284

100 0.060832023620605 0.042212018966675

1000 0.62564301490784 0.032389879226685

10 000 0.65919399261475 0.050267934799194


Interpreatation of the resutls: Clearly MongoDB outperfomed MySQL by far in WRITE


operations, however MySQL slightly outperformed MongoDB in READ operations.Results are

somewhat inconsistent which is probalby due to other processes running on the operating

system. However, MySQL performance could also be affected by the storage engine used. In the

experiments InnoDB is used as a storage engine, but other storage engines are available such as

MyISAM, ARCHIVE, BLACKHOLE, etc. Each storage engine would perform differently because it

provides different features.

The code used in the experimentation can be found in Appendix B

6. Conclusion

The primary objective of the project was achieved, namely the new system is up and running at

the Experimental Solar Plan. The separation of concerns – data collection and analyses provides

robust interface for continues data collection and on‐demand data analyses. Specifically, the

fetcher (data collection) and be run at any given point in time for as long as needed; on the

other hand, the data analysis works independently so data could be analyzed in both real time

and/or at a later point. The system could be extended by adding robust data analyses

functionality, but the current graphics interfaces could be reused. Also, in experiments showed

that MongoDB has better overall performance than MySQL , since the MySQL writes are very

slow compared than MongoDB writes, but MySQL reads were slightly faster. However,

MongoDB provided better optimal performance.


7. References:

[1] Brown University Data Managment and Reserach Group, "MongoDB Database Design Research,"

Brown University, [Online]. Available: http://database.cs.brown.edu/projects/mongodb/.

[2] Canonical Ltd., "Home | Ubuntu," [Online]. Available: http://www.ubuntu.com/. [Accessed 20 04

2013].

[3] 10gen, Inc., "MongoDB," 10gen, Inc., [Online]. Available: http://www.mongodb.org/. [Accessed 10

04 2013].

[4] Wikipedia, "Row (database)," 17 05 2013. [Online].

[5] K. Chodorow and M. Dirolf, MongoDB The Definitive Guide, O'Reilly, 2010.

[6] T. Talov, "Solar Plant Data Logging," 2013.

[7] Campbell Scientific. Available: http://www.campbellsci.com/cr1000.

Appendix A – Experimentation <?php echo''; echotime_elapsed(); // Configuration $dbhost = 'localhost'; $dbname = 'test'; // Connect to test database $m = new Mongo("mongodb://$dbhost"); $db = $m->$dbname; // Get the users collection $c_users = $db->users; // Insert this new document into the users collectionf for($i=0; $i<1000; $i++){ // $c_users->save($user) $users = array( 'first_name' => $i, 'last_name' =>'Fan'.$i ); $c_users->save($users); } echotime_elapsed(); $con=mysqli_connect("localhost","root","","foo"); // Check connection if(mysqli_connect_errno()) { echo"Failed to connect to MySQL: " . mysqli_connect_error(); } for($i=0; $i<1000; $i++){ mysqli_query($con,"INSERT INTO persons (first_name, last_name) VALUES ('Peter', 'Griffin')"); } echotime_elapsed2().'\n'; echotime_elapsed(); echo'\n'; $db = $m->selectDB('test'); $collection = new MongoCollection($db, 'users');

echotime_elapsed(); mysqli_query($con,"SELECT * FROM persons"); echotime_elapsed2(); functiontime_elapsed() { static$last = null; $now = microtime(true); if($last != null) { echo''; } $last = $now; } functiontime_elapsed2() { static$last = null; $now = microtime(true); if($last != null) { echo''; } $last = $now; }