improved novel based ant colony clusters for fast ... › hub › 2018-120-6 › 4 › 340.pdf ·...

Improved Novel Based Ant ColonyClusters for Fast Execution of Large

Datasets

S K Chaitanya Rudraraju1, Chalasani Srinivas2

and Prathipati Ratna Kumar31,2Assistant Professor in Computer Science

and Engineering Department,SirCRRCoE, Eluru.

[email protected],[email protected]

3Associate Professor in Computer Scienceand Engineering Department,

VSL Engineering College, [email protected]

Abstract

Technology is taking its hike and data related tomemory is increasing day so on. Today world is de-pendent on larger repositories much more. Everyorganization is growing with employees and theirnature of work. It is difficult for them to deal withpersistent data which is available or scatted on dif-ferent databases. Problem is with larger datasetsavailable on repositories. To deal with situationslike this, paper focus on improved ant colony clus-ters on larger databases or repositories. Techniqueis that to collect repositories, collect tables, andcollected tables are purified. We get the columnswhich are to be computed. Collected data should becleaned and then compressed for calculations. The

International Journal of Pure and Applied MathematicsVolume 120 No. 6 2018, 4675-4692ISSN: 1314-3395 (on-line version)url: http://www.acadpubl.eu/hub/Special Issue http://www.acadpubl.eu/hub/

4675

result analysis can be a graph or original data. Thisprocess is related to operational big data.

Key Words : Databases, Repositories, Compressed,Datasets, Operational Big Data.

1 INTRODUCTION

Technology is growing rapidly; changes towards technology arealso changing in the same manner. Data plays a key role in thetoday’s world. Everywhere data is scattered in mobile phones,cameras, laptops, computer systems, embedded systems and so on.From 2003 to 2011, the data produced is 6 million giga bytes. Sameamount is generated in every 10 days now. If we scatter all disks,it will occupy one national airport. This rate of growth in datais increasingly daily. Big data are two words faced by the world.Big data is not a normal data; it is really, really big in data. Bigdata is a collection of large datasets that cannot be computed byusing traditional approaches. Big data is not a course, subject oranything; it is an ocean, which involves various tools, techniquesand frameworks. Today’s world is depending upon the memory as-pects. Every where people collect videos and photos. They havetheir own database for their personal and private works. Evenpeople store their childhood memories. Now a day’s people speakabout clouds and memories very much. They store each and everydata in memory. Dataset corresponds to a single database table orsingle statistical data matrix. A table contains columns and rows.Rows represent data related to a column. There are different typesof data that comes under big data. Data related to helicopter, airflights and jets. It captures voices of the flight crews, recordingof microphones and earphones performance information of the air-craft, identifying prior errors related to the flight. Social mediawebsites such as facebook, instagram and twitter hold informationand sharing of views posted by millions of the people across theglobe. Stock exchange is a market in which securities are boughtand sold. The stock exchange holds large data information aboutselling and buying of the shares of different companies. Decisionsmade on shares by vary customers. Power grid data holds informa-tion consumed by a particular house (node) with respect to a base

International Journal of Pure and Applied Mathematics Special Issue

4676

station. Railway system which contains data related to reservation,train data from source to destination (time tables), catering, plat-form, and so on. Andhra Pradesh State bus transport data whichincludes model, capacity, distance, availability of a vehicle, reser-vation, employees and so on. Search engine which retrieves lot ofinformation from varies different databases.

Same way companies and industries are expanding their prod-ucts, employees, branches and soon. They are spreading theirwings. They are increasing their products and sales. When mem-ory is huge obviously speed falls based on the device. Even in worstcases the speed in retrieving the data should not fall and should beaccurate.

When a company or industry grows into bigger, the need forsearching and retrieving data also grows. Very fast retrieval orsearch should be done on large data sets. Obviously searching largedata consumes more time. The user who requires information fromdata sets has to wait for long time to see result. Sometimes it maygo to hours or a day(s). This depends on available data sets, therestorage system and other parameters. Before search or retrieval wehave to position the data sets. If data sets contain non-numeric datait consumes more memory. Numeric data can be compressed [1].Searching non-numeric data is a task. Lot of memory is requiredto process the data. In best case we may except result within oneor two records, medium case we may find result at half way and inworst case at the end of the records. When we come to numericdata we can divide them into clusters. Each cluster is of fixed sizeand allowed to store data in it as ant after ant (i.e., Ant Colony).

Big data technologies should provide more accurate analysis,which may lead to concrete decision-making resulting in greateroperational efficiencies, cost reduction and reduced risks for thebusiness. To exhibit the power of big data, one should requirean infrastructure that can manage and process huge volumes ofstructured and unstructured data in real time and can protect dataprivacy and security. There are various technologies in the marketfrom different vendors including Amazon, IBM, and Microsoft etc.,to handle big data. While looking into the technologies that handlebig data, we examine the following two classes of technology:


4677

• Operational Big Data

• Analytical Big Data

This paper focuses on operational big data. This includes sys-tems like MongoDB that provide operational capabilities for real-time, interactive workloads where data is primarily captured andstored. MongoDB is one the database type under the NoSQL ban-ner. MongoDB is built on architecture of collections and docu-ments. Documents comprise sets of key value pairs and are thebasic unit of data in MongoDB. MongoDB stores data in flexible,JSON like documents, meaning fields can vary from document todocument and data structure can be changed over data. The doc-ument model maps to the objects in your application code, makingdata easy to work with. Adhoc queries, indexing and real time ag-gregation provide powerful ways to access and analyze your data.This process collects data from lot of tables and transforms datatables to schemas that are suitable for operational big data. Miningprocess is a length process. Data cleaning and prepare data cubeand then goes to mining process. For mining it requires two pro-fessionals, software professional and business analyst to formalizeresult of the data. It is just for GB data. Now we are in TB infuture this may go to PB or EB or ZB or YB. Searching or comput-ing this large data from collection of data sets is a hurdle. To solvethis problem we have to go for better and accurate result executingmodel. One of the models is Ant Colony Clusters [1].

2 LITERATURE REVIEW

Konda Sreenu, et. al. introduced Ant Colony Cluster, a newmodel, but no implementation about ant colony clusters. Modelfocus on compression of the data sets for execution. Paper focuson methods like columnar databases, run length environment tech-nique and ant colony clusters. No implementation related to themodel.

O. A. Mohamed Jafar, et. al. introduced Ant-based clusteringis a clustering technique aims at the unsupervised classification ofpatterns in different groups. These are so many algorithms devel-oped for solving numerical and combinational optimization prob-


4678

lems. Among them are swaron-based algorithms. It is a conven-tional clustering technique. It focus on the behavior of ants.

Wei Gao, introduced clustering analysis that are used in manydisciplines and applications. The ant-colony clustering algorithm isa swaron-intelligent method used in various cluster problems. Thatinspired by the behavior of ant colonies. Author focus on the modelof ant colony algorithm, but no implementation about ant colonycluster algorithm.

Ling chin, et. al. introduced an artificial ants sleeping model(ASM) and adaptive Artificial Ants Clustering Algorithm (AYC)are presented to solve clustering problems. They are simulationbased solution but author does not focus implementation and noidentification of experimental results.

Bao-Jiang Zhoo, introduced ant colony clustering algorithm foroptimistically executing large data sets. This algorithm was im-plemented and tested on several real data sets. Time is the maindrawback of the paper.

Urszula Boryczka, introduced bio-inspired techniques for clus-tering algorithms received attention for performance & stability tomake mature tools for data mining. Author introduced algorithmsand procedures to the new approach. But no focus on implemen-tation.

Xiaoyong Liu, et. al. introduced clustering algorithm to solvethe unsupervised clustering problem. Ant colony optimization (ACO)is used to solve combinational optimization problem which is basedon stochastic best solution kept - EsaCC. This model cannot com-pute with present large data sets.

Jinbiao Wang, et. al. introduced ant-based clustering algo-rithm, Ant colony clustering have different characteristics like clus-tering, high clustering accuracy, irregular cluster shapes and so on.All these topics are related to clustering algorithm. No implemen-tation details.

Hong Jiang, et. al. introduced basic model of ant colony cluster-ing algorithm. Purposed IACC an improved ant colony clusteringshows better performance than LF method and it should be im-proved.

Dong Liyan, et. al. introduced micro-clustering, traditional antcluster algorithm is not accurate. Ant Colony clustering algorithmis based on swarm intelligence. This algorithm proposed a new


4679

policy of picking and dropping objects. Author proves that antcolony clustering is better algorithm based on swarm intelligencethan traditional colony algorithm in terms of efficiency. It is basedon dividing the similar objects into one cluster.

3 EXISTING SYSTEM

Memory plays a key role in today’s world. Memory size isgrowing every day. Retrieving and storing data is also becominga task. Large companies have more records related to their em-ployees, sales, billing and so on. Today’s world is dependent uponcomputer systems. They want their search or results very fast ina finger tips. To get fast results on data sets, we have to focuson data mining. In data mining, data is divided into two parts,non-numeric and numeric data. Non-numeric data occupies morememory and almost worst case of search because there will no re-peated data. All data will occupy memory. Numeric data we cansave memory by applying compressing techniques. Data should becollected and then it must be cleaned, make schemas preparationfor collected data, compress the data into ant colonies. One afterthe other, place data for computing. Expose the result in a reportor graph format. Final result will contain report or a graph.

We design and implement a pattern to solve business user re-quirements without data mining expert and business analyst. Theprocess of mining will take time to display results. But businessuser wants result very fast. Data is collected from different reposi-tories and other sources. Collected data cleaning process will carryon. Data cleaning means removing unwanted non-key attributes,records with null values. After cleaning records are made to be antcolony cluster. Each cluster can be of 100 data records size or chunkwith fixed size. If data block is more than 100 records, then it isadded to next block. Data in blocks can be compressed. Data withnumeric can be compressed but alphabets cannot be compressed.There are so many algorithms like Bitmap Index, Look-and-saySequences, Run-length limited, LZW or run-length encoding forcompression. It will be is easy to accumulate large data. Thereshould be large memory enough to hold data collected from var-ious sources. Results are made to display for user. Data can be


4680

displayed in reports form, picture form, text form or any other waywhich can be understood by business user.

Algorithm

Step 1: Collect data from various database sources.

Step 2: Data cleaning process. Remove unwanted non-key at-tributes, columns with null values.

Step 3: There are three tables in a database. Table - 1 Employee,Table - 2 Product and Table - 3 Dept.

Step 4: Arrange the data in columnar database manner.

Columnar database have data blocks for 100 record col-umn.

Normally table is given in the following manner.


4681

Step 5: Apply aggregate functions or any other calculation meth-ods over the data.

Block Diagram

The existing method is called Ant Colony Cluster Methodas shown in below figure 1 (Fig. 1). The data blocksare made to available like ants row. Data blocks can beexpanded to their right side. It can grow to maximum asmuch memory is available. After data blocks are prepared,based on business user requirements we can use aggregatefunctions like SUM, COUNT, AVG, MAX, MIN.

Figure 1: Ant Colony Cluster Method


4682

4 PROPOSED SYSTEM

In this research work, we designed and implemented a patternto solve business user requirements without data mining expert andbusiness analyst. The business user can use his system to calculatethe results fast. Data is collected from huge repositories and largedatabases. Focus of this paper is on numeric data only. Extensionof this paper can consider non-numeric and numeric with index.Model is given in below figure 2.

Algorithm for Improved Ant Colony Cluster

Step 1: Data from various repositories

Example:- Database related to employee or university.

Step 2: Collect data tables.

Example:- EMP, DEPT and SALGRADE Tables.

Step 3: Clean Data for executing values.

Example:- Get only numeric columns.

EmpDatabase-Emp1 Table→Sal

EmployeeDatabase-EMP Table→SAL

EMPdatabase- E Table→Salary

Step 4: Apply compressing techniques.

Example:-

Step 1-Collect similar data into blocks and compress data

(0.0001)5, (0.1)15, (0.2)7

Step 5: Frame them as columnar clusters.

Example:- From EMP tables, we want to calculate salariespaid to all employees who are located at different placeson globe for one organization.

Ant Colony Clusters:- Q1, Q2, Q3, ....... Qn are clusterswith sum aggregation on a sal column.


4683

Step 6: Arrange frames as ant by ant manner.

Step 7: Apply any Aggregate function.

A1→∑(S1 + S2 + S3 + .....)

A2→∑(S1 + S2 + S3 + .....)

A3→∑(S1 + S2 + S3 + .....)

Step 8: Result is exposed in a form of a report or a graph.

Figure 2: Improved Ant Colony Cluster


4684

5 IMPLEMENTATION

Carrefour is a super market located in 11,935 areas at vari-ous different companies with 3,84,151 employees around the world.Again there is room for more vacancies. In future, organization isplanning for more stores and more employees. For every calculationit will take more time. Here there is scope for data mining, big dataanalytics and cloud techniques over billing system, payroll, items,items searching, item purchase, items stock and many more. Sup-pose if we want to calculate the annual salaries paid to employeesfrom last four years.

According to algorithm, collect the tables from various brancheslocated at various places.

Step 1: Collect various databases located at different branches.

Step 2: Collect all tables required for query.

Step 3: Collect all the numeric columns from the tables by prepar-ing them or mining as shown in below figure (Fig. 3). Inthe table we SAL’ as column which data is divided by10000. So that the resulting value many hold the datasize. If required we can divide by more digits. It is de-cided by experts before the mining process take place. Ineach table the columns may be named different like empid,empno, salary, sal, empsal and so on. We have to mergedata with different field names to one field SAL’. We canclean data for executing process.

Figure 3: Preparing data for mining


4685

Step 4: When we want to perform calculations like search, theprocess starts from left to right by visiting each columnsdata. This will consume lot of time as shown in belowfigure (Fig. 4).

Figure 4: Which consume lot of time

Columnar is a process where it will select the data of saidcolumn from initial data to last data, vertically. We canconsume lot of time and can have data clearly and accu-rately as shown in below figure (Fig. 5).

Figure 5: For time Consuming

Step 5: Apply data compression techniques to reduce memory ex-


4686

ercises as shown in figure below (Fig. 6). By data com-pression we can save 25% of the memory.

Figure 6: Data Compression Techniques

Step 6: Arrange the all table columnars as ant like structure oneafter the other as shown in below figure (Fig. 7). Bypreparing data in ant colony order, it is easy to executedata one after the other very fast and easy manner.

Figure 7: Ant Colony

Step 7: Apply aggregate functions on the available data. Acquireddata may sometime go beyond the available data size that


4687

is available. The result plan should be performed prior tostart of the plan.

Step 8: The result can be exhibited in a graph when the result hasmultiple values or result can be exhibited as original databy multiplying the value that we divided at step 3.

Code Implementation

The code can be implemented by creating considering weka,RapidMiner, KNIME, Orange, SPSS Modeler, Oracle Data Mining,ELKI, Rattle GUI, Statistica, Apache Mahout, Teradata warehouseand soon or we can implement software externally for data mining.The procedures that should be considered as follows.

When we compare result with normal data mining techniqueand with improved ant colony cluster is given result analysis inbelow figure (Fig. 8).


4688

Figure 8: Result Analysis

6 CONCLUSION

From past history onwards there are lots of changes in mem-ory and there retrieval exercises. This paper focus on one of thetechniques to recover result analysis from larger databases or repos-itories that can accommodate lots of data. We have lots of dataand require results as fast as possible. Paper focus on such kindof techniques to deal with larger repositories. It is an improvedversion of ant colony cluster technique applied on supermarket likeCarrefour. The result analysis gives satisfaction above normal tech-nique. There is scope for more improvement.

References

[1] “Ant Colony Clusters for Fast Execution of Large Data sets”,Konda Sreenu, Dr. B Raja Srininivasa Reddy, Unpublished.

[2] “Ant-based Clustering Algorithms: A Brief Survey”, O. A. Mo-hamed Jafar and R. Siva Kumar, ISSN: 1793-8201, IJCTE,Vol. 2, No. 5, October, 2010.

[3] “Improved Ant Colony Clustering Algorithm and its Per-formance Study”, Wei Gao, Computational Intelligence andNeuroscience, Volume 2016, Article ID: 4835932, 14 pages,http://dx.doi.org/10.1155/2016/4835932.


4689

[4] “An Adaptive Ant Colony Clustering Algorithm”, Ling Chen,Xiao-Hua XU, Yi-Xin Chen, Proceeding of the third inter-national conference on Machine Learning and Cybernetics,Shanghai, 26-29 August 2004, IEEE.

[5] “An Ant Colony Clustering Algorithm”, Bao-Jiang Zhao, ISSN(Print): 2160-133X, (Electronic), ISSN: 2160-1348, 29 Oct’2007, IEEE Xplore.

[6] “Ant Clustering Algorithm”, Urszula Boryczka, ISBN: 978-83-60434-44-4, Pages 377-386, Intelligent Information System,2008.

[7] “An effective Clustering Algorithm with Ant Colony”, Xiaoy-ong Liu, Hui Fu, Journal of Computers, Vol. 5, no. 4, April’2010.

[8] “An Ant Colony Clustering Algorithm Improved from ATTA”,Jinbiao Wang, Ailing Tu, Hongwei Huang, 1875-3892, Pub-lished by Elsevier BV. Selection and/or peer-review under re-sponsibility.

[9] “An improved ant colony clustering algorithm”, Hong Jiang,Qingsong Yu, Yu Gong, ISSN (Print): 1948-2914, (Electronic):1948-2922, IEEE Xplore: 18th Nov’ 2010.

[10] “Ant Colony Clustering Algorithm Based on Swarm Intel-ligence”, ISBN (Electronic): 978-1-4799-2809-5, CD-ROMISBN: 978-1-4799-2810-1, IEEE Xplore: 24 July’ 2014.

[11] “Approach for Developing Business Statistics using Data Webusage mining”, G VS CH S L V Prasad, Malapati Sri RamaLakshmi Reddy, Kuntam Babu Rao, Chodagam Suresh Ku-mar, ISSN (Print): 2319-2526, Volume - 1, Issue - 2, 2012.

[12] “Approach for Developing Business Statistics using weblogmining”, Konda Sreenu, Dr. B. Raja Srinivasa Reddy, ISSN(Online): 2347-2812, Volume - 2, Issue - 11, 2014.

[13] “Providing Security and efficiency in Attribute based Datasharing with novel technique”, Gudipudi Ravi Chaitanya Ku-mar, Prathipati Ratna Kumar, ISSN (Online): 2347-2820, Vol-ume - 3, Issue - 12, 2015.


4690

[14] “Preprocessing a Unsupervised Approach for Web Usage min-ing”, Addanki Ramya, Konda Sreenu, P Ratna Kumar, ISSN:2252-8784, Volume - 1, No. 2, December 2012.

[15] “Data Method and Mining Techniques for better Business Or-ganization”, Adabala Hanumantha Rao, Prathipati Ratna Ku-mar, ISSN: 2347-2820, Volume - 3, Issue -12, 2015.

[16] “A Freight Stabilization Model Based on Cloud Segregating forthe shared cloud”, S Venkata Siva Satish, P Ratna Kumar,ISSN (Online): 2278-5841, ISSN (Print): 2320-5156, IJRCCT,Volume - 3, Issue - 11, November - 2014.

[17] “De-Duplication of Citation Data by Genetic ProgrammingApproach”, Seetalam Divya Manusha, Valiveti Karthik,Prathipati Ratna Kumar, IJRAET, ISSN(Online): 2347-2812,Volume - 1, Issue - 3, 2013.

[18] “Slicing: A New Approach to privacy preserving High dimen-sional Data Publishing”, IJSETR, ISSN (Online): 2319-8885,Volume - 2, Issue - 15, November - 2013.

[19] “Fast Searching Technique on Sequential Large Databasesin an organization”, Samparthi V Suresh Kumar, MatchaGanesh Babu, J S V Gopal Krishna, Suryadevra Mohan BabuChowdary, Konda Sreenu, B Homer Benny, Prathipati RatnaKumar.


4691