using clickthrough data to optimize search result...

152
USING CLICKTHROUGH DATA TO OPTIMIZE SEARCH RESULT RANKING An evaluation of clickthrough data in terms of relevancy and efficiency ANVÄNDNING AV CLICKTHROUGH DATA FÖR ATT OPTIMERA RANKNING AV SÖKRESULTAT En utvärdering av clickthrough data gällande relevans och effektivitet Bachelor Degree Project in Informatics 30 ECTS Spring Term 2017 Anton Paulsson Supervisor: András Márki Examiner: Henrik Gustavsson

Upload: others

Post on 10-Jun-2020

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

USING CLICKTHROUGH DATA TO

OPTIMIZE SEARCH RESULT RANKING An evaluation of clickthrough data in terms of

relevancy and efficiency

ANVÄNDNING AV CLICKTHROUGH

DATA FÖR ATT OPTIMERA RANKNING

AV SÖKRESULTAT En utvärdering av clickthrough data gällande

relevans och effektivitet

Bachelor Degree Project in Informatics

30 ECTS

Spring Term 2017

Anton Paulsson

Supervisor: András Márki

Examiner: Henrik Gustavsson

Page 2: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

Abstract

Search engines are in a constant need for improvements as the rapid growth of

information is affecting the search engines ability to return documents with high

relevance. Search results are being lost in between pages and the search algorithms are

being exploited to gain a higher ranking on the documents. This study attempts to

minimize those two issues, as well as increasing the relevancy of search results by

usage of clickthrough data to add another layer of weighting the search results. Results

from the evaluation indicate that clickthrough data in fact can be used to gain more

relevant search results.

Keywords: [Search Result Reorganization, Clickthrough Data, Information Retrieval,

Apache Solr]

Page 3: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

Table of Contents

1 Introduction ........................................................................................................ 1

2 Background ........................................................................................................ 2

2.1 Search Engines ...................................................................................................... 2 2.1.1 Apache Lucene ............................................................................................................... 2 2.1.2 Apache Solr ..................................................................................................................... 3 2.1.3 Sphinx ............................................................................................................................. 4

2.2 Search Algorithm .................................................................................................... 4

2.3 Clickthrough Data ................................................................................................... 4

2.4 Stack Overflow Data Dump ..................................................................................... 5

3 Problem .............................................................................................................. 6

3.1 Aim ......................................................................................................................... 7

3.2 Hypothesis .............................................................................................................. 7

4 Method ................................................................................................................ 8

4.1 Systematic Literature Review .................................................................................. 8

4.2 Approach ................................................................................................................ 9

4.2.1 Relevancy Testing ........................................................................................................... 9 4.2.2 Performance Testing ....................................................................................................... 9 4.2.3 Research ethics............................................................................................................. 10

5 Implementation ................................................................................................ 11

5.1 Systematic Literature Review ................................................................................ 11

5.2 Progression........................................................................................................... 11 5.2.1 Solr Implementation ...................................................................................................... 11 5.2.2 MySQL Database .......................................................................................................... 12 5.2.3 Search platform ............................................................................................................. 12 5.2.4 Clickthrough Data Query Parser Plugin ........................................................................ 13 5.2.5 Clickthrough Data Algorithm ......................................................................................... 14

5.3 Pilot Study ............................................................................................................ 16

6 Evaluation ........................................................................................................ 17

6.1 The Study ............................................................................................................. 17

6.1.1 Efficiency Evaluation ..................................................................................................... 17 6.1.2 Relevancy Evaluation .................................................................................................... 21

6.2 Analysis ................................................................................................................ 24

6.3 Conclusion ............................................................................................................ 26

7 Concluding Remarks ....................................................................................... 27

7.1 Summary .............................................................................................................. 27

7.2 Discussion ............................................................................................................ 27

7.3 Research Ethics .................................................................................................... 28

7.4 Future Work .......................................................................................................... 29

References .............................................................................................................. 30

Page 4: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

1

1 Introduction

The need for better and more precise search engines is constantly increasing. Users of the

World Wide Web (WWW) are in large need of being able to access information efficiently,

but at the same time as the need increases, so does the information. According to Chua

(2012) the average increase of information available on the Internet is as large as 23% per

year, which requires search engines to get better at handle the information that is being

indexed. To achieve this, search engines have to adapt to the major information increases by

using different search algorithms. The most commonly used algorithms to rank documents

are PageRanking and Hyperlink Induced Topic Search, but both of them are out-dated and

easy to exploit. (Pawar & Natani, 2014)

Joachims (2002) suggests that usage of clickthrough data would help the search engines to

rank the search results. By looking at what documents are being visited the most and what

documents that the user tends to choose. Users have different opinions and thoughts behind

the choices they make, which can’t be collected, but the choices they make can however be

collected. This information could be used to potentially increase the relevancy of the search

results that are being displayed to the user by adding a new layer of document scoring. One

of the major issues with the large amount of information that search engines have to handle

is the problem that relevant documents gets lost in between different pages of results.

Osiński (2003) explains how most documents are never visited due to users not navigating

further in to the page numbers. In theory, this could be improved by the system that

Joachims (2002) suggests.

In this study an experiment is performed to be able to evaluate if clickthrough data can be

used to increase relevancy of search results without impacting the efficiency of the search

engine in such a large manner that it becomes inefficient. An algorithm based on

clickthrough data is developed and implemented in Apache Solr (Apache Solr, 2017). The

search engines index is then populated with a dataset provided by Stack Overflow (Stack

Overflow, 2014). To be able to record the clickthrough data, a minor search platform is

developed. The search platform is developed in Php, JavaScript and HTML, the clickthrough

data is then stored in a MySQL database to allow the algorithm to retrieve the data. To

evaluate the efficiency of the algorithm, a version of Apache Solr with the clickthrough data

algorithm is compared to an original version of Apache Solr. An addition of a quantitative

survey is used to be able to treat the aspect of relevancy, as relevancy is not something that

can be measured without human judgement.

A pilot study was performed where 20 human testers were asked to perform seven different

queries on the search platform. The objective was to see if there were any major time

differences between an original version of Apache Solr and a version of Apache Solr with the

implemented clickthrough data algorithm. The pilot study showed that on average the

clickthrough data algorithm added an additional 5 milliseconds to the full request time.

Page 5: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

2

2 Background

Information is no longer something that is stored in file rooms and cabinets; it is now

something that is available all over the word through the World Wide Web. Fukumura,

Nakano, Harumoto, Shimojo, & Nishio (2003) describe how information is being moved

over to the digital world and leaving the papers behind.

Databases are a typical way of storing information, it allows information to be sorted and

easily found. The majority of the population on earth does however not have the knowledge

of how to access information directly from a database, therefore some sort of graphical

interface is needed. This is one of the reasons that the demand for high quality search engines

is so large. Search engines enables anyone to access the information on the internet no matter

what their computer knowledge is. Fakhraee & Fotouhi (2011) describes how the importance

for search engines grows as the information available in databases grows and that the

current amount of data is already massive.

2.1 Search Engines

Search engines are tools with absolutely no technical skill required, which allows users to be

able to access information available on the World Wide Web. It allows users to be able to

locate information through usage of keywords and sometimes full-text search.

There are several different types of search engines. Two of the most regular types of search

engines are general search engines and internal search engines. General search engines are

the kind of search engines that are used every day by the public. Google is the most used

general search engine according to “NetMarketShare”(2017). It is used by 85% of all desktop

users and 97% of all mobile and tablet users. Google is a website where users are allowed to

search through all of the information that Google has indexed in their system and receive a

link directly to the source of the information.

Internal search engines allow users to be able to find information that is stored locally on a

website. Bian, Li, Yue, Lei, Zhao & Xiao (2015) explains that an internal search engine is

used to be able to locate information that general search engines are not able to find, or at

least puts the result so lowly ranked that it is very hard to find. For example a university

usually has its own website, where documents and information is released to the staff and

students. To be able to find this information and documents quickly an internal search

engine is one of the most rapid ways of accessing it. While using a general search engine to

find the same data could take several minutes or hours of browsing before it has been found.

A search engine needs to handle three different problems by three different parts according

to Singh, Hsu, Sun, Chitaure & Yan (2005). The first part that has to be created is an index

where the information will be stored. The second part consists of creating an algorithm that

can retrieve the information. The last part is creating an interface where the information is

presented after it has been retrieved.

2.1.1 Apache Lucene

Apache Lucene is an open source text search engine created and open sourced by Doug

Cutting in 2000. Lucene consists mainly of its index and the possibility to query for

Page 6: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

3

documents in the index; it has the functionality to index documents as well as an API to

allow remote querying of the documents. It is based on Java but allows for usage in several

different programming languages, such as Python or .Net. (Balipa & Balasubramani, 2015)

Lucene offers several different features that allow for creation of a complete search engine,

such as an inverted index to be able to efficiently retrieve documents, a large set of text

analysis components, a query syntax that allows for a large number of query types and a

scoring algorithm for weighting of the documents. (Smiley & Pugh, 2011)

2.1.2 Apache Solr

Apache Solr is an open source enterprise search server created by the Apache Software

Foundation. The server is written in Java and built on top of Apache Lucene. Apache Solr

also contains a web administration interface out of the box (see Figure 1). Nagi (2015)

describes it as “a web application that can be deployed in any servlet container” and lists

the addition of functionality that Solr brings that Lucene does not have. The functionalities

that are added in Solr are:

XML and JSON APIs

Hit highlighting

Faceted search and filtering

Geospatial search

Caching

Near real-time searching of newly indexed documents.

Web administration interface

The additional functions allow Solr to become more versatile, compared to Lucene, which

can lead to more research that include Solr.

Figure 1 - Apache Solr Web Administration Tool

Page 7: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

4

2.1.3 Sphinx

Sphinx is an open source search server created by Sphinx Technologies Inc. A web

administration interface is available through Sphinx tools (see Figure 2). It is a full text

search server created to with performance and relevancy in mind (Sphinx, 2017). The system

is mostly built in

Figure 2 - Sphinx Tools Admin Panel

2.2 Search Algorithm

The user searches for information by asking the search engine a question, either by simple

words or by usage of full-text questions. The search engine then analyses the question and

try to find information that is considered related to the search question. Depending on how

the search algorithm is built, it will select different “answers” to the question. The search

engine then replies with the most relevant information. Pawar & Natani (2014) explains that

one of the key elements to a good search engine is a ranking algorithm. As the relevancy of

the search result depends on the ranking algorithm. According to Pawar & Natani (2014) the

two most common algorithms used to rank web pages are (1) PageRanking algorithm and (2)

HITS (Hyperlink Induced Topic Search).

2.3 Clickthrough Data

When a user completes a search, he or she is presented with a list of alternatives to choose to

continue reading about. In most cases a user does not automatically choose the first available

option after completing a search. Depending on how relevant the title of the option “feels”

like for humans, they choose different options. Clickthrough data can be used to be able to

change the ranking of the options available, depending on which options that the users tend

to choose (Joachims, 2002).

Clickthrough data can be used to store which options a user chooses, as well as in what order

the options was chosen. This can be used to create a new layer in the search engines

weighting layer, so that the results can be re-ordered depending on the provided data that

Page 8: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

5

the clickthrough system uses. This means that the search engine would be able to improve

depending on how much it is being used. In theory, the more users that are using the system

the faster it will improve. Joachims (2002) explains how he implemented a system with an

equal solution. It allowed the search results to be re-ordered, so that the most relevant and

interesting documents would slowly move upwards in the ranking system and more relevant

information could easier be found by the users of the search engine.

2.4 Stack Overflow Data Dump

Stack Overflow is an internet forum in the Stack Exchange network containing thousands of

topics regarding technical questions and answers. It is used by more than 6.7 million users

and has more than 40 million visitors every month. (Stack Overflow, 2017)

Every quarter of the year Stack Exchange publishes a dump of all of the user-contributed

content on Stack Exchange since 2009. The dump is also licensed under Creative Commons

BY-SA 3.0 which allows anyone to use it as long as the credit and posts are still linked back

to the original posts or users. Each site that is in the Stack Exchange network is also

downloadable individually. (Stack Exchange, 2017)

Page 9: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

6

3 Problem

The amount of information handled by search engines increases for every year, at the same

time as the demand of information availability by the users also increase. This requires

search engines to get more efficient at handling the information at the same time as it has to

get better at sorting the information that the users are requesting. Chua (2012) estimates the

growth of the information available on the Internet by an average of 23% per year.

The presentation of data in a search engine is one of the major issues that needs to be

addressed, as Osiński (2003) explains that a lot of the information that users want access to

disappears in between the massive amount of information that exists. It especially becomes a

problem as the information is usually displayed in an ordered list that can be split in to

multiple pages. It creates the problem that users need to choose a very specific search

question to be able to find relevant information from a search result.

The algorithms used to rank documents in search engines are very limited. Pawar & Natani

(2014) talks about how search engines tend to only use two different implementations of a

ranking system. They explain that most usual ways of ranking documents are too restricted

to be able to gain an actual good way of judging what documents are relevant to a specific

search-term. They also explain that the PageRanking algorithm is easy to exploit as it is

based off of how many terms that can be associated to the search query. This allows the

document creator to abuse the algorithm by entering terms that automatically puts a

document at a higher rating, even though it may not contain as much relevant information as

another documents. In the same way as PageRanking can be exploited, so can Hyperlink

Induced Topic Search. It evaluates the relevancy depending on how many times the

document has been referred to, which could be hyperlinks for example. As social media posts

or similar are a part of the indexing systems as much as any other documents, it is simple to

create a large amount of hyperlinks to a document through social media. This also allows

documents to gain a higher score by excessively creating social media posts that refers to the

document.

According to Joachims (2002) a larger problem for the clickthrough data algorithm was that

the system he created was not able to detect differences between a regular user performing a

search query and a user “spamming” the system. This created a problem with so-called “false

relevancy”, as hyperlinks that was selected a large amount of times, would slowly get better

ranking in the search engine. This was the effect of basing the ranking only on which results

were chosen the most.

Page 10: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

7

3.1 Aim

The goal of this research is to evaluate if clickthrough data can efficiently be used to increase

the relevance of search results. As user experience is dependent on the time it takes for a

search result to be displayed, it has to be efficient, otherwise the majority of users will not

continue to use it. By using an existing search engine that is open source and combining it

with an algorithm that can rescore documents depending on clickthrough data, tests can be

made by installing and configuring an original version of the same search engine. This will

allow for evaluation of the algorithm both my measuring time differences as well as

performing user tests.

Research question: Will a clickthrough data algorithm allow search engines to

efficiently display more relevant search results?

To be able to derive an answer to this question 4 objectives needs to be addressed:

1. Gain knowledge of the domain by examining existing literature.

2. Design an experiment and implement algorithm to evaluate clickthrough data.

3. Perform experiment and collect data.

4. Analyse the results and present conclusion.

3.2 Hypothesis

The hypothesis of this study argues that based on existing theory a clickthrough data

algorithm can be created to display search results with a better precision to the search term

compared to the original algorithm in the search engine. It also argues that the algorithm

will add a minor delay to retrieve the results, but that the delay is low enough for the search

engine to be efficient.

Page 11: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

8

4 Method

Performing evaluation of software engineering consists of three major empirical techniques

according to Wohlin et. al. (2012). They define the three major empirical techniques as

surveys, case studies and experiments. A survey consists of collecting data from humans or

about humans to be able to draw conclusions from that data. A case study consists of

reviewing data by looking at an existing method or tool at a corporation or an organization.

An experiment consists of testing the system in a controlled environment and manipulates

one factor or variable of the studied setting.

The chosen method for this study is experiment with a quantitative approach. It is most

suitable for the study as it depends on being able to test performance differences through

manipulation of the algorithm in a precise and systematic way. Just as Wohlin et. al (2012)

mention as the main definition of when an experiment is suitable.

Experiments are launched when we want control over the situation and want

to manipulate behaviour directly, precisely and systematically.

Wohlin et. al., 2012, pp. 16

In addition to the experiment a data collection will be completed through a quantitative

survey, to determine the relevance of the search results in a similar approach as Clarke et. al

(2008) suggests in their framework. This is required to be able to answer the research

question, as it depends on being able to evaluate the relevance of the search algorithm at the

same time as evaluating the efficiency of the algorithm.

An alternative to performing an experiment, a case study could be possible to execute to be

able to answer the research question. As a case study would let the algorithm be used in its

intended real-world application, which could provide a more accurate answer to the research

question. A large problem with performing a case study is the issue of finding an appropriate

organization to implement the system at or one that is already using a similar

implementation of relevancy weighting, which also could create legal complications. Another

aspect to take in to account is also that an experiment allows the testing to be performed in a

more controlled environment compared to performing a case study according to Wohlin et.

al. (2012).

4.1 Systematic Literature Review

To be able to gain background knowledge of the field a systematic literature review (SLR)

will be performed. The SLR will be based on a set of papers gathered from IEEE Xplore and

the ACM portal using a set of search queries (see Appendix A). To able to filter the papers

depending on their relevance to the field a set of criteria’s was defined both for inclusions

and exclusions (see Appendix A). Jalali & Wohlin (2012) showed that if an SLR is executed

and performed in an efficient way, that this method is a very exceptional way of gaining a

deeper understanding and an improved knowledge of the field. It also allows researchers to

be able to not stumble in to the same problems as earlier research has already experienced.

Page 12: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

9

4.2 Approach

In this experiment a clickthrough data algorithm will be implemented together with a search

engine and tested. The install of the search engine will be indexing the posts from the dataset

that Stack Overflow is providing as a test dataset with more than 30 million entries. The

testing of the algorithms is split in to two parts. (1) Measurement of relevancy differences

between a basic install of a search engine and an install of a search engine combined with the

Clickthrough Data Algorithm (CDA). (2) Measurement of performance differences between a

basic install of the search engine and an install of the search engine combined with the CDA.

4.2.1 Relevancy Testing

Joachims (2002) proposes a framework to be able to test the relevancy difference with and

without a CDA. It lets users see two different search result lists and lets the user choose

which list that the user found most relevant to what he or she was looking for. Joachims

(2002) is using a very similar way of judging relevancy to the framework that Clarke et. al.

(2008) presents. It is based on the human assessor selecting a binary answer to each search

result, if it is either (a) relevant or (b) irrelevant, to the search question asked.

The most effective way of measuring relevancy on keyword search systems is conducting a

survey according to Joachims (2002) and Clarke et. al. (2008). As relevancy is not

something that can be defined, it is important to get a broad audience to contribute to the

survey. As humans tend to have different definitions of what is relevant, it can depend on a

lot of different factors; for example it can depend on how many times the search words is

included in the document found, it could be dependent of how the document is interpreted,

hence why it is so important to gain a broad selection of people to judge the relevancy.

4.2.2 Performance Testing

Wohlin et. al. (2012) explains how measurements are used to be able to make judgments

based on facts instead of intuition. When there are measurements you are able to compare it

and allow the data to be analyzed, find out where the bottlenecks are as well as being able to

determine if resources are being used efficiently.

To be able to measure the efficiency of the clickthrough data algorithm, measurements of the

time differences with and without the CDA will be performed. To be able to measure the

differences, two installations of Apache Solr will be run on identical systems. The

installations will be using identical indexes of the Stack Overflow dataset, the only difference

will be that one will be running with the CDA, and one will not. Both of the systems will be

measured during the testing to be able to answer the following questions:

1. How large is the time difference in retrieving the search result for the user with and

without the CDA activated?

2. How long does it take for the CDA to run?

3. How does the CDA impact the search engine’s efficiency?

Page 13: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

10

4.2.3 Research ethics

The dataset gathered from Stack Overflow is completely sanitized from any personal

information to keep the information anonymous according to Stack Exchange (2014).

An important part to make sure that the survey data does not contain any personal

information. To be able to know that the entries are unique some sort of generation of a

unique identifier needs to be created, that could be based on IP-addresses or something

similar. It is also very important to be able to decide what could be considered as personal

information, as per definition personal information is any kind of data that could be derived

to a specific person.

Another aspect to take in to account is to be sure that the persons that are included in the

study knows and allows usage of the data that they create. If these terms are not accepted,

the data will not be used nor collected. Participants are allowed to quit the survey whenever

they want and any data that the participant has created will not be used.

To be able to treat the aspect of reliability and repeatability in software testing, all of the

implementations, configurations, results, software, hardware specifications and any other

resources that are relevant to the experiment, will be presented as appendixes to allow

anyone to repeat the experiment and verify the study.

Page 14: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

11

5 Implementation

5.1 Systematic Literature Review

There are many ways of implementing a system that allows similar evaluation of a search

engine and different algorithms, as there are several programming languages that can be

used to accomplish very similar results. To be able to create a test environment that is

suitable for this study, an availability approach was chosen. All of the software,

programming languages and programming platforms chosen, was all open source in order to

be able to keep the study as repeatable as possible.

Selecting a search server to use was not easy; all different systems have a lot of pros and

cons. It was narrowed down to two options, Apache Solr or Sphinx Search. However, the

final choice was Apache Solr. The first reason that Solr was chosen over Sphinx, was due to

the licensing. As the Apache2 license does not apply any requirements that could limit the

usage of the system, whereas Sphinx is licensed under GPLv2 which does apply conditions

depending on its usage. Another reason for choosing Solr over Sphinx is the possibility to

store very large indexes, as the Stack Overflow dataset consists of more than 30 million

documents. According to Khabsa, Carman, Choudhury & Lee Giles (2012) Solr can hold an

index as large as 3 billion documents, but there is in fact no documented limit for how large

of an index Sphinx Search can hold.

Inspiration for how to configure Solr was taken from the book Apache Solr 3 Enterprise

Search Server by Smiley & Pugh (2011). It contains information of how to set up Solr, lists a

large amount of problems that may occur and how to handle them.

To build the web platform and online survey environment, JavaScript will be used for front-

end and PHP/MySQL for the backend. The combination of PHP and MySQL allows for a

secure connection between the two, as PHP as integrated support for a secure connection to

MySQL. All three techniques are open-source, which allows for simple and easy usage.

The Stack Overflow dataset will partially be used to build the document base in the search

engine. The information that will be used is the posts created by the users of Stack Overflow,

which consists of more than 30 million documents.

5.2 Progression

This section explains the different design choices made and the implementation of scripts,

servers and other related parts of the system. The parts are listed in the order that they were

created or installed and any obstacles that may have occurred. The hardware specifications

of the server that all of the server-side code and software ran on can be found in Appendix

S.

5.2.1 Solr Implementation

Apache Solr was installed with the newest version available at the start of the research, which

was version 6.4.3 at the 1st of March 2017. Configuration of the Solr core was performed as

per Appendix B, Appendix C and Appendix D. This allowed usage of the built in Data Import

Handler.

Page 15: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

12

A problem that occurred during the indexing of the dataset was a lack of heap memory for

the Java Virtual Machine. Smiley & Pugh (2011) mentions that when scaling up Solr for

larger indexes it is common that the default heap memory for the JVM is simply not enough

and that it may need some fine tuning to minimize the risk of getting an

“OutOfMemoryException”. The solution to the problem was to allow the JVM to use more

memory during the import by adding the “-m” parameter when launching Solr. This allowed

Solr to use a total of 6 gigabytes of heap memory instead of the default 512 megabytes and

solved the errors that occurred during the importation of the dataset.

5.2.2 MySQL Database

The MySQL database is where all of the clicks performed by users are being stored. It stores

information regarding the search query, which document that has been clicked and the

amount of times the document has been clicked on. The database table configuration staid

the same until late parts of the experiment. The only changes that were made (see Figure 3

and 4) were an increase of the size of the “full-term” column from 40 characters to 80

characters, an increase of the “term” column size, from 20 characters to 30 characters as well

as changing the default character set to UTF-8 to match the character set of the search

platform. The reason behind changing the sizes was to be able to fit larger words and search

terms in to the columns. The table configuration is also listed in Appendix G.

Figure 3 – Initial database table configuration of the Clicks table

Figure 4 – Final database table configuration of the Clicks table

5.2.3 Search platform

The main application of the system is the search platform (see Figure 5). It is the very thing

that ties each part of the system together. It is developed in PHP and JavaScript, where PHP

is used for querying for documents in Solr, displaying the documents to the user as well as

exchanging information with the MySQL database, the connection between MySQL and PHP

Page 16: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

13

can be seen in Appendix F. JavaScript handles Ajax requests to call for storing the

clickthrough data in the database.

Figure 5 - Search Platform

At the start of the project the idea was to use a PHP extension for Solr (Php.Net 2016), which

is an object-oriented library for communication between PHP and Solr. However, the library

is quite out-dated as it is several versions behind the development of Solr and it lacks

support for some of the features that are offered in the Solr API. Instead a connection to the

Solr API was used as presented in Appendix J.

The collection of clicks is performed through JavaScript (See Appendix F) that listens for a

click event to occur, then it checks if the user has the cookie for the same hyperlink & search

query combination. If it does, then no clickthrough data is recorded, but if it doesn’t it sends

an Ajax request that sends the data to the MySQL database. The JavaScript code is presented

as Appendix K and the PHP file that handles the Ajax request is presented as Appendix E.

5.2.4 Clickthrough Data Query Parser Plugin

Whenever the parameter “rq={!cda cdaWeight=2 cdaDocs=500}” is used while requesting

documents from the Solr API, the Clickthrough Data Query Parser Plugin is called. It takes

care of searching the documents that matches the query and if the documents do, they will

be re-scored by the Clickthrough Data Algorithm. The parser also allows for a set of optional

Page 17: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

14

parameters to be added, weight of the new score and the amount of documents to be re-

scored. The weighting is performed by multiplying the “click score” with the chosen weight.

If no weight or number of documents is chosen, the algorithm defaults the weight at 2.0 and

rescores the top 200 documents in the result list. When all of the documents have been

checked and possibly re-scored, the list of documents gets sorted depending on the score of

each document and the list is sent back to where the request was called from. The full source

code is presented in Appendix I.

5.2.5 Clickthrough Data Algorithm

When the Clickthrough Data Query Parser finds a document that matches the query the

Clickthrough Data Algorithm rescoring is called. The algorithm then checks if the document

has any recorded clicks, if the document does not, no added score is returned. If it does, the

algorithm then checks if the clicks match the query and if they do, additional scoring is

added depending on the rate of the clicks (as seen in Figure 6). The full source code is

presented in Appendix H.

Figure 6 - Clickthrough Data Algorithm score method

Page 18: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

15

Figure 7 – First running version of the Clickthrough Data Algorithm score method

Figure 8 - Clickthrough Data Algorithm Constructor

A large issue in the early versions of the clickthrough data algorithm was added delay in each

query. As shown in Figure 7, the reason behind this issue was that it performed a request to

the click storage table in the database each time the score method was called. At that stage,

each query would take longer than 10 seconds to retrieve, which just was not efficient

enough. The API request was moved to the constructor of the Clickthrough Data Algorithm

class (see Figure 8), as well as the API was redesigned to respond with clicks depending on

the query term instead of responding with clicks depending on the ID of the document. By

changing the structure of the Clickthrough Data Algorithm class, a single request would

respond with the clicks of all of the documents for the query instead of one request per

document. This made a large impact on the time it takes to perform a query. Instead of every

query taking more than 10 seconds to retrieve, it would be faster than 1 second each time.

Page 19: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

16

5.3 Pilot Study

A pilot study was performed on the clickthrough data algorithm, comparing the speed that

the documents are presented with and without the algorithm. A set of 20 testers used the

search functionality to perform seven different queries. The objective was simply to find out

if there was a time difference between using the added algorithm or not, as well as to check

how big the gap was, if there was any.

Chart 1 - Pilot study average request time in seconds

The measurements of the request speeds showed very promising results, it showed that there

was infact a minor difference between using the algorithm and not using it, as shown in

Chart 1. On average the Clickthrough Data Algorithm added an additional 0.05 seconds

delay to collect the information compared to a request without the algorithm. However that

result is gained with a very small test set of clicks compared to how it would be if the general

public would have populated the clickthrough data table. Which proves that the algorithm

has to have a larger set of clicks to encounter to be able to make a fair assumption of it’s

performance.

A minor issue found during the testing, was that the system was missing a mechanism to find

out if any document had been rescored according to the clickthrough data algorithm. This

could have added a minor time difference, as the system would try to sort the array of

documents even if there was no actual difference between before and after the clickthrough

data algorithm had completed its rescoring.

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Seco

nd

s

Test ID

Average request Time (s)

Clickthrough Data Algorithm Average Regular Search Average

Page 20: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

17

6 Evaluation

6.1 The Study

The experiment was conducted in two parts. The first part was to evaluate the efficiency of

the CDA by comparing a version of Solr with the CDA to a version of Solr without the CDA.

The second part was to evaluate if the CDA would increase or decrease the relevancy of the

search results by using an online survey.

6.1.1 Efficiency Evaluation

Search Term Index Size Number of Results

Windows Install Small (5,000 documents) 25

Java Get Time Small (5,000 documents) 10

JavaScript Cookies Small (5,000 documents) 5

Windows Install Medium (50,000 documents) 134

Java Get Time Medium (50,000 documents) 36

JavaScript Cookies Medium (50,000 documents) 10

Windows Install Large (1,000,000 documents) 5157

Java Get Time Large (1,000,000 documents) 1268

JavaScript Cookies Large (1,000,000 documents) 350

Table 1 – Number of results for each search term depending on index size

To be able to determine if the clickthrough data algorithm was affected by differences in

amount of search results, a collection of three queries were chosen (see Table 1). The reason

that just those 3 search terms were used was because that they all differ in amount of search

results in all of the 3 index sizes. The search term “Windows Install” had the most amount of

results, “JavaScript cookies” had the fewest amount of results and “Java Get Time” was

somewhere in between the other two.

Page 21: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

18

Search Term Index Size Search Algorithm

Windows Install Small (5,000 documents) Clickthrough Data Algorithm

Windows Install Medium (50,000 documents) Clickthrough Data Algorithm

Windows Install Large (1,000,000 documents) Clickthrough Data Algorithm

Windows Install Small (5,000 documents) Original Solr

Windows Install Medium (50,000 documents) Original Solr

Windows Install Large (1,000,000 documents) Original Solr

Java Get Time Small (5,000 documents) Clickthrough Data Algorithm

Java Get Time Medium (50,000 documents) Clickthrough Data Algorithm

Java Get Time Large (1,000,000 documents) Clickthrough Data Algorithm

Java Get Time Small (5,000 documents) Original Solr

Java Get Time Medium (50,000 documents) Original Solr

Java Get Time Large (1,000,000 documents) Original Solr

JavaScript Cookies Small (5,000 documents) Clickthrough Data Algorithm

JavaScript Cookies Medium (50,000 documents) Clickthrough Data Algorithm

JavaScript Cookies Large (1,000,000 documents) Clickthrough Data Algorithm

JavaScript Cookies Small (5,000 documents) Original Solr

JavaScript Cookies Medium (50,000 documents) Original Solr

JavaScript Cookies Large (1,000,000 documents) Original Solr

Table 2 – Parameter array for efficiency tests.

A set of 18 parameter combinations were measured to be able to determine the efficiency of

the clickthrough data algorithm at 3 different index sizes. The complete set of parameter

combinations can be seen in Table 2. Each of the parameter combinations were run for 200

iterations, which resulted in a total of 1800 results with the CDA activated (see Appendix

M) and a total of 1800 results without the CDA activated (see Appendix N). To be able to

see how the efficiency has been affected in Apache Solr, there are 3 major aspects to

evaluate:

1. How the algorithm affects the request speed depending on how many results there

are for the search term.

2. How the algorithm affects the request speed depending on the index size of the

search engine.

3. How the algorithm affects the request speed in general.

The request speed is the time that it takes between a request was sent from the search

platform to Solr until the search platform has retrieved the search results and presented

them to the user.

Page 22: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

19

Chart 2 - Comparison chart of request speeds for all of the parameter combinations with

standard deviation

To address the first Chart 2 shows the average of the measurement results of the request

time for each of the search terms listed in Table 1. Each of the search words have been

iterated 200 times for each index size. As no other testing was performed on any search term

with as low amount of search results, the reason behind the low request time for the search

term “JavaScript Cookies” is unsure. An educated guess would be that it is due to the low

amount of results for that search term in the small index, which was a total of 5 search

results.

Chart 3 – Comparison of the request speed depending on the index size with and without

the Clickthrough Data Algorithm

Page 23: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

20

By combining the results for the for search terms on each index size a more fair assumption

of the differences in request speed between the index sizes can be made. Chart 3 displays

the average request speed for each index size with standard deviation. The measurements

used in Chart 3 are a combination of the 3 different search terms that was run on the same

index size, combining a total of 600 iterations.

Chart 4 – Full report of 1800 executions of the Clickthrough Data Algorithm

Chart 4 displays all of the measurement iterations that were run on the version of Apache

Solr with the clickthrough data algorithm. It shows that the execution speed of the CDA,

which is the time it takes from when the query parser plugin (see Chapter 5, Section

5.2.4) receives the request to start rescoring the documents of the query and until it has

finished processing all documents and possibly, re-sorted them.

Page 24: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

21

Chart 5 - Comparison of the full page load time of the search platform, with and without the

Clickthrough Data Algorithm

The full page load time of the search platform is the time it takes from when a user has

connected to the webserver through a web browser and until the page has been fully loaded.

In the case shown in Chart 5 the page loaded also includes a query to Solr and the retrieval

of the search results, the “Other” part is all of the material that is loaded that has no ties or

connections to the search engine itself. The measurement in that chart consists of the

average value for each respective part of the page load, which has been iterated 1800 times

for each version of Solr.

6.1.2 Relevancy Evaluation

A quantitative survey was created to be able to investigate if users preferred the ranking that

the clickthrough data algorithm created or not. A total of 100 individuals with some prior

knowledge of programming were asked to partake in the survey. The survey was closed after

45 respondents had finished it. The reason behind asking for prior programming knowledge

was due to the fact that most of the possible search results are related to programming as

Stack Overflow is an online programming community (Stack Overflow, 2017). The computer

and programming experience for each user is shown in Chart 6.

Page 25: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

22

Chart 6 - Years of professional computer/programming experience for all survey

participants. Each bar represents a participant of the survey.

After each participant had filled in their experience they were presented with the search

results from a randomly selected search term, displaying the results for both the original

version of Solr and the version of Solr with the clickthrough data algorithm next to each

other. Then the participants were instructed to select the result list that they would say

provided the best answer to the search term (see Figure 9). Each participant performed this

task twice with different randomly selected search terms until every search term had been

evaluated 15 times. Chart 7 displays all of the selections that were done by the participants

of the survey.

Page 26: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

23

Figure 9 - Survey User Interface

Chart 7 – Survey result list selections

Page 27: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

24

6.2 Analysis

The difference in request speed is the major interest area as that is what can be affected the

most by the clickthrough data algorithm. The request speed is the time it takes from the

moment a query has been sent to the API of Apache Solr from the search platform and until

the data has been retrieved and displayed to the user. After performing a total of 3600

iterations of the measurements, 1800 iterations with and 1800 iterations without the

clickthrough data algorithm, the average difference between the request speeds lowered by

more than half compared to the pilot study. Chart 8 displays the average request speed for

both the version of Solr with and the version without the clickthrough data algorithm. As

shown in Chart 8 the time difference is small enough to barely be able to notice a difference

between the two versions. On average the version of Solr running with the clickthrough data

algorithm was 0,028 seconds (2.8 ms) faster than the original version of Solr, even after a

much larger set of clicks compared to the pilot study was recorded.

Chart 8 - Comparison of request Speed with and without the clickthrough data algorithm

with standard deviation

When comparing the average request speeds between different index sizes regardless of what

search term was used, it is a clear how little the index size matters (see Chart 3). Comparing

the standard deviation for all of the tests, it is visible that all except the original version of

Solr with a large index intersect at approximately 0,34 seconds. This proves that on average

the index size does not affect the general efficiency of Solr. No correlation could be found

between the clickthrough data algorithm execution time and the index size, see Appendix N

for full report of measurements.

By comparing the full page load (see Chart 5) of the original version of Solr and the version

of Solr with the clickthrough data algorithm it is clearly visible that the only time difference

Page 28: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

25

between them are a few milliseconds. The average time it takes for the clickthrough data

algorithm to be fully executed is 0,00304 seconds (see Chart 4 for all 1800 iterations) and

no other components of Solr should be affected by the clickthrough data algorithm as it is

implemented as one of the last components to be executed in Solr. Which is clearly presented

in Chart 5 as the time difference between the two versions is barely separable.

Chart 9 - Average survey result list selection

The survey data shows that in most cases the clickthrough data algorithm increases the

relevance of the search queries performed (see Chart 9). The only search term where the

original version of Solr is chosen over the version with the clickthrough data algorithm is the

term “php shorthand” (see Chart 7). However, by examining the full survey results (see

Appendix O) a correlation between participants with low experience and participants

receiving the search term “php shorthand” is seen in multiple cases, which could have

affected the results of that specific search term, which would also affect the average list

selection.

Page 29: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

26

6.3 Conclusion

By examining Chart 3 and 4 it is clear that the index size of Solr does not affect the speed of

retrieving results in a large manner. This proves that the clickthrough data algorithm can be

used at any size without causing any major loss of time. The only exclusion in the

measurements performed in this study was when a search term had a lower number of hits,

it would on average have a lower request speed when the size of the index was of the smaller

size.

The measurements shown in Chart 3, 4 and 8 indicates that the clickthrough data

algorithm does not add any major delay during the retrieval of search results in such a large

manner that the system becomes inefficient. On average the supposedly added delay was

low enough (less than 4 milliseconds) for the version of Solr with the clickthrough data

algorithm to have a lower average full page load time than the original version of Solr. It was

actually 0,2 milliseconds faster on average. The standard deviation in Chart8 of both

versions of Solr intersect, which shows that on average there are no difference between the

two versions, which also proves that the algorithm is not affecting the efficiency of the search

engine.

When comparing the search results in both the version of Solr with the clickthrough data

algorithm and the original version of Solr, the survey data shows that the algorithm is

increasing the relevancy of the search results, as shown in Chart 9. Out of the 90 queries

that participants performed, even with several participants who had little or no professional

experience of programming, in 67.78% of the search queries the participant preferred the

version of Solr with the added algorithm. This indicates that the algorithm returns a scoring

of documents that is according to the survey data, a bit more accurate.

The hypothesis was approximately correct, as it was possible to create an algorithm based on

clickthrough data using the existing theory and literature. By evaluating the algorithm it can

be concluded that the algorithm increased the relevance of the search results at the cost of a

minor delay. However, most online testing tools consider the page load time of the search

platform to be in the top percentiles of the sites registered, for example GTmetrix (2017)

places the search platform in the top 2% fastest pages of the 182 million pages that has been

analysed.

Page 30: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

27

7 Concluding Remarks

7.1 Summary

The goal of the study was to evaluate if clickthrough data could efficiently be used to increase

the relevance of search results. This was investigated by performing an experiment where an

algorithm was developed and implemented in the search engine Apache Solr. The hypothesis

of the study was:

“The hypothesis of this study argues that based on existing theory a clickthrough data

algorithm can be created to display search results with a better precision to the search

term compared to the original algorithm in the search engine. It also argues that the

algorithm will add a minor delay to retrieve the results, but that the delay is low enough

for the search engine to be efficient.” (Chapter 3, section 3.2)

This study concludes that clickthrough data can be used to efficiently increase the relevance

of search results by implementing a clickthrough data algorithm in Apache Solr. The

efficiency evaluation indicates that the algorithm on average is fully executed in 3,04

milliseconds, which affects the full page load by an average of 0,38%. The survey shows that

the algorithm provides an increased level of relevance in the search results, as in 67,78% of

the queries that were completed, the participants chose the version of Apache Solr with the

algorithm above the original version of Apache Solr. Both evaluations prove the strength of

the hypothesis at the same time as it proves that clickthrough data can be used to efficiently

increase the relevancy of search results.

7.2 Discussion

By examining the results of the experiment an important factor that plays an important role

are the search terms that are used and its respective clicks. What could have been affecting

the search results are the clicks that have been recorded during the study and how they could

have affected the choices that were made by the participants in the survey. The clicks that

has been used to calculate scores for the documents in the survey are the clicks that has been

recorded (see Appendix R for full list of clicks) by using the during the study as well as the

clicks that were recorded during the pilot study instead of manipulating the clicks manually.

The order of the results was drastically changed and the score changes that were added by

the clickthrough data algorithm were rather large. This was most likely affected by the

weight chosen during the query to Solr, as the amount of clicks that were relatively low, a

higher weight were chosen to be able to see any differences in the ordering of the search

results. As Joachims (2002) describes, the idea is that the system is created to be able to re-

order the search results that are presented for better or worse, depending on what

documents the users tend to choose. A higher weight could affect the order of the search

results as much in a positive light as well as in a negative light, as all of the documents that

have been chosen by other users gain as much scoring depending on the recorded clicks no

matter if the document is a relevant or irrelevant one, as long as the user visited the page

where the document is listed.

As all of the measurements that were performed were executed in the web browser Google

Chrome, no conclusion regarding how each web browser could have affected the possible

Page 31: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

28

differences in speed can be made. However, the important variables of the measurements

were all located on the server side, as both the request to Solr and the algorithm are both

located on the server side. This means that the web browser should not have affected the

results, but is there is still not enough information to be able to draw a conclusion.

The goal of the research is to evaluate if clickthrough data can efficiently be used to increase

the relevance of search results. The study accomplished the goal and was able to derive an

answer to the research question by performing a systematic literature review to address the

first objective. The second and third objective was addressed by implementing the

clickthrough data algorithm, performing the experiment and conducting the survey. The last

and final objective uses the results from the previous objectives to address the hypothesis

and answers the research question.

Other research mostly uses clickthrough data to accomplish automatic personalization of

systems. The only related work that could be found was the work that Joachims (2002)

performed an experiment with a similar usage of clickthrough data. The main difference

between this study and the study that Joachims (2002) performed was the addition of a

Support Vector Machine, which there simply was not enough time and resources to apply to

this study. However, the result of his work matched the conclusion of this study.

The study also raises questions regarding the usage of clickthrough data and how reliable it

can be. Even with an implemented anti-spam functionality it can be exploited to raise the

ranking of a search result by usage of scripts and bots. It is easy to keep the click storage

clear of invalid clicks in a controlled environment, but in a system open to the general public,

it is very hard to get rid of all the exploitations that could be performed. It is also unsure how

safe it is to collect information regarding the behaviour of the users in a system, especially on

a global level. Even if it the information cannot be tied to a specific user, it is uncertain if it is

ethically correct. There is a major risk that the data could be used to violate the users by

storing other information regarding the users. For example the way that Google stores

information regarding the content that the users browse and make suggestions for the user

depending on that information. As the implementations of clickthrough data can be seen in

this study, it is possible to see that the used information is not violating the users

information, but in other similar implementations that may not be the case.

7.3 Research Ethics

As the evaluation of the relevance differences was performed by using an online survey it was

very important to consider the personal integrity of the participants. To secure the integrity

of each user, a random hash was created for each user as an identifier. No information that

can be considered as personal information has been stored and the questions asked was not

focused on each participant as an individual, but rather focused on the relevant factors that

could affect the results that the participants would generate by participating in the survey.

The issue of using a small number of participants in the survey can be argued against the

credibility of this study. The most optimal circumstance would include a much larger

number of participants to gain more accurate statistical results.

Wohlin et al (2012) describes randomization as one of the general principles for designing

tests, as randomization allows tests to become as general as possible. The same principle was

Page 32: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

29

applied to the survey by randomizing the search term for every test from a set of search

terms (see Appendix O) until each search term was evaluated 15 times, as well as

randomizing the position of the result lists.

Validity and reliability was taken in to consideration during the experiment, which are two

factors that affected the large number of measurement iterations on both the original version

of Solr and the modified version of Solr. As well as the main reasons that the participants of

the survey was asked for their experience in regards to computers and programming.

Potential errors may have occurred throughout the study, but none of them has been

purposely added. No measurements or survey has been removed or altered to assure the

reliability of the results. All of the information that was collected from external sources has

been referenced to the original source.

7.4 Future Work

The results from this study indicated that clickthrough data can be used to affect the

relevance of search results. However, there is a lot of room for improvements that can be

made to gain a more precise and accurate ordering of the results. If given more time to

compare additional search terms and recording a much larger set of clicks, a more general

result could be accomplished. There is also the issue that the only indexing system that was

evaluated was Apache Solr. To be able to know if the algorithm is applicable to any other

indexing system, further research would be needed where a similar algorithm would be

implemented in other indexing systems, like Sphinx (Chapter 2, Section 2.1.3) for

example.

In this study the storage of clickthrough data is located in a separate MySQL database. If the

storage of clicks would instead be implemented in the search engine, it could potentially save

time for the algorithm and allow it to become more efficient. At the same time it would allow

the stored information to become more secure, as the risk of hackers breaching a database

exists. While it is more complex to breach a search engines index or internal storage as long

as the search engine is configured with proper security.

The system itself has not been stress tested and would most likely not gain the same results

when dealing with larger amounts of traffic. It would be interesting to observe data of the

same algorithm being used in an environment with several thousands of users, which was

limited in this study as there was not nearly enough resources. This could be accomplished

by performing a case study for example. On a larger scale some fine tuning of the weighting

system and a better solution of how to handle the amount of clicks versus the score of the

document would most likely be required. As in this experiment there is a certain point where

the original score that Solr gives to each document does no longer matter, due to the scoring

of the algorithm being a larger number than the original score of the document.

Clickthrough data could most likely be applied in many other domains to make the overall

experience a bit more relevant and compute different types of scorings. It could for example

be applied to navigation systems, to be able to present the user with differently scored

routes. As usually there is not just a single route for each destination, users have to choose

which route that they would like to take. Clickthrough data could be used to be able to

retrieve options from different users who has been starting at the same position and had the

same destination and present a route that is most used by other drivers, pilots or captains.

Page 33: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

30

References

Balipa, M., & Balasubramani, R. (2015). Search Engine using Apache Lucene. International

Journal of Computer Applications, 127(9), pp. 27-30.

Bian, Z., Li, Y., Yue, T., Lei, P., Zhao, D. & Xiao, Y. (2015) An information Update

Method Towards Internal Search Engine. In 12th Web Information System and

Application Conference (WISA), Jinan, Shadong, China, 2015, pp. 211-216.

Chua, C. (2012). A user interface guide for web search systems. In Proceedings of the 24th

Australian Computer-Human Interaction Conference, OzCHI ’12. ACM, New York,

NY, USA, pp. 76–84.

Clarke, C. L., Kolla, M., Cormack, G. V., Vechtomova, O., Ashkan, A., Buttcher, S. &

MacKinnon, I.(2008) Novelty and diversity in information retrieval evaluation. In

Proceedings of SIGIR, 2008, pp. 659–666.

Coffman, J., Weaver, A.C. (2010) A framework for evaluating database keyword search

Strategies. In Proceedings of the 19th ACM International Conference on Information

and Knowledge Management, CIKM ’10. ACM, New York, NY, USA, pp. 729–738.

Fakhraee, S & Fotouhi, F. (2011) Effective Keyword Search over Relational Databases

Considering Keywords Proximity and Keywords N-grams. In 22nd International

Workshop on Database and Expert Systems Applications. 2011 Toulouse, France,

pp. 190-194.

Fukumura, S., Nakano. T. , Harumoto, K., Shimojo, S. & Nishio, S. (2003) Realization of

Personalized presentation for digital contents based on browsing history. In IEEE

Pacific Rim Conference on Communications Computers and Signal Processing,

2003, Victoria, B.C, Canada. 2003, pp. 605-608 vol.2.

GTmetrix [WWW Document](2017) URL https://gtmetrix.com/ (accessed 05.22.17)

Jalali, S., Wohlin, C. (2012). Systematic Literature Studies: Database Searches vs. Backward

Snowballing. In: Proceedings of the ACM-IEEE International Symposium on

Empirical Software Engineering and Measurement, ESEM ’12. ACM, New York, NY,

USA, pp. 29–38.

Joachims, T. (2002) Optimizing search engines using clickthrough data. In ACM SIGKDD

international conference on Knowledge discovery and data mining. Edmonton,

Alberta, Canada, July 23 - 26, 2002. New York, NY, USA: ACM. pp.133-142.

Khabsa, M., Carman, S., Choudhury, S. R., & Giles, C. L. (2012, August). A framework for

bridging the gap between open source search tools. In SIGIR 2012 Workshop on

Open Source Information Retrieval, 2012, pp. 32-39.

Page 34: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

31

Nagi, K. (2015) Bringing Search Engines to the Cloud using Open Source Components. In 7th

International Join Conference on Knowledge Discobery, Knowledge Engineering

and Knowledge Management (IC3K), Lisbon, 2015, pp. 116-126.

NetMarketShare [WWW Document] (2017). URL http://www.netmarketshare.com/

(accessed 02.03.17).

Osiński, S. (2003) An algorithm for clustering of web search results. University of

Technology, Poland.

Pawar, S. G. & Natani, P. (2014) Effective utilization of page ranking and HITS in significant

information retrieval, International Conference for Convergence for Technology-

2014, Pune, 2014, pp. 1-6.

Php.Net [WWW Document](2016) URL http://php.net/manual/en/book.solr.php

(accessed 04.04.17)

Smiley, D., & Pugh, D. E. (2011). Apache Solr 3 Enterprise Search Server. Packt Publishing

Ltd. Birmingham, United Kingdom. pp. 7-26, 75-98, 119-122 , 147-150, 334.

Sphinx [WWW Document] (2017). URL http://sphinxsearch.com/about/sphinx/

(accessed 02.08.17)

Singh, R., Hsu, Y.-W., Sun, W.-C., Chitaure, D., Yan, L. (2005). Rethinking the Presentation

of Results from Web Search, in: IEEE International Conference on Multimedia and

Expo, Amsterdam, pp. 1492–1495.

Stack Overflow [WWW Document] (2014) URL http://stackoverflow.com/company/about

(accessed 02.08.17)

Stack Exchange [WWW Document] (2017) URL http://stackoverflow.blog/2014/01/stack-

exchange-cc-data-now-hosted-by-the-internet-archive/ (accessed 02.08.17)

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B. & Wesslén, A. (2012).

Experimentation in Software Engineering. Springer, Heidelberg, Berlin. pp. 9-18, 94

Page 35: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

I

Appendix A - Search Queries

Search queries used on IEEE Xplore, Google Scholar and The ACM Digital Library to collect

information from research papers:

“Clickthrough Data”

“Apache Solr”

“Sphinx Search”

“Search Engine” AND “Clickthrough Data”

“Search Engine” AND “Relevancy Algorithm”

“Search Engine” AND “Efficiency”

Systematic Literature Review Criteria’s

Inclusions

Books, papers and technical reports containing:

Empirical findings.

Relevance improving systems for search engines.

Exclusions

Studies that did not contain any information regarding Clickthrough Data or any

form of relevance increasing system.

Studies that was found irrelevant to search engine optimization.

Page 36: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

II

Appendix B - Schema.xml

Configuration added in the Solr/server/core_name/conf/schema.xml file at row 121.

<field name="_version_" type="long" indexed="false" stored="false"/>

<field name="Id" type="string" indexed="true" stored="true" required="true"/>

<field name="PostType" type="int" indexed="false" stored="false" />

<field name="ViewCount" type="int" indexed="false" stored="true"/>

<field name="Content" type="text_en" indexed="true" stored="true"/>

<field name="Owner" type="int" indexed="false" stored="true"/>

<field name="Title" type="string" indexed="true" stored="true"/>

<field name="Tags" type="string" indexed="true" stored="true"/>

<field name="AnswerCount" type="int" indexed="false" stored="true"/>

<field name="FavoriteCount" type="int" indexed="false" stored="false"/>

<uniqueKey>Id</uniqueKey>

Page 37: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

III

Appendix C - Solrconfig.xml

The edited configuration in the Solr/server/core_name/conf/solrconfig.xml file, at rows 87,

739 and 1198.

Row 87:

<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-

.*\.jar" />

Row 739

<requestHandler name="/dataimport"

class="org.apache.solr.handler.dataimport.DataImportHandler" startup="lazy">

<lst name="defaults">

<str name="config">data-config.xml</str>

</lst>

</requestHandler>

Row 1198

<schemaFactory class="ClassicIndexSchemaFactory"/>

Page 38: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

IV

Appendix D - data-config.xml

XML structure used by the dataImportHandler to index the posts from Stackoverflow.

<dataConfig>

<dataSource type="FileDataSource" encoding="UTF-8" />

<document>

<entity name="page"

pk="Id"

processor="XPathEntityProcessor"

stream="true"

forEach="/posts/row/"

url="C:\StackOverflow\Posts.xml"

transformer="RegexTransformer,DateFormatTransformer">

<field column="Id" xpath="/posts/row/@Id"/>

<field column="PostType" xpath="/posts/row/@PostTypeId"/>

<field column="CreationDate" xpath="/posts/row/@CreationDate"

dateTimeFormat="yyyy-mm-dd'T'hh:mm:ss.SSS"/>

<field column="Score" xpath="/posts/row/@Score"/>

<field column="ViewCount" xpath="/posts/row/@ViewCount"/>

<field column="Content" xpath="/posts/row/@Body"/>

<field column="Owner" xpath="/posts/row/@OwnerUserId"/>

<field column="Title" xpath="/posts/row/@Title"/>

<field column="Tags" xpath="/posts/row/@Tags"/>

<field column="AnswerCount" xpath="/posts/row/@AnswerCount"/>

<field column="FavoriteCount" xpath="/posts/row/@FavoriteCount"/>

</entity>

</document>

</dataConfig>

Page 39: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

V

Appendix E - Clickthrough data collection

PHP code that handles the insertion and update of the click count in the database.

<?php

require 'dbconn.php';

$id = getPostAJAX("ID");

$fullTerm = getPostAJAX("fullTerm");

$terms = explode(" ", $fullTerm);

if ($id == "null" || $fullTerm == "null" || $terms == "null") err("Missing Form

Data: (ID/FullTerm/Terms)");

//Control if the term-id combination exists in the database

foreach($terms as $term){

try{

$q1 = "SELECT Clicks FROM Clicks WHERE linkID = :linkID AND Term = :Term";

$clicks = $pdo->prepare($q1);

$clicks->bindParam(':linkID', $id);

$clicks->bindParam(':Term', $term);

$clicks->execute();

$clickCount = $clicks->fetchColumn();

} catch (PDOException $e) {

err("Error!: ".$e->getMessage()."<br/>");

die();

}

//If the line exists in the database, update it.

if($clickCount >= 1){

try{

$updateClicks = $clickCount + 1;

$q2 = "UPDATE Clicks SET Clicks = :Clicks WHERE Term = :Term

AND linkID = :linkID";

$update = $pdo->prepare($q2);

Page 40: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

VI

$update->bindParam(':linkID', $id);

$update->bindParam(':Term', $term);

$update->bindParam(':Clicks', $updateClicks);

$update->execute();

} catch (PDOException $e) {

err("Error!: ".$e->getMessage()."<br/>");

die();

}

//if it doesn't, insert it.

}else{

try{

$q3 = "INSERT INTO Clicks (linkID, Clicks, fullTerm, Term)

VALUES(:linkID, '1', :fullTerm, :Term)";

$insert = $pdo->prepare($q3);

$insert->bindParam(':linkID', $id);

$insert->bindParam(':fullTerm', $fullTerm);

$insert->bindParam(':Term', $term);

$insert->execute();

} catch (PDOException $e) {

err("Error!: ".$e->getMessage()."<br/>");

die();

}

}

}?>

Page 41: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

VII

Appendix F - dbconn.php

The php code allowing for a PDO database connection to be created.

<?php

function getPostAJAX($param) {

if(isset($_POST[$param])){

if($_POST[$param] === "0"){

$ret = "0";

}else if(empty($_POST[$param])){

$ret = "null";

}else{

$ret = htmlentities(urldecode($_POST[$param]));

}

}else{

$ret = "null";

}

return $ret;

}

function dbConnect() {

global $pdo;

try {

$pdo = new PDO("mysql:dbname=".DB_NAME.";host=".DB_HOST,DB_USER,DB_PASSWORD);

$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

} catch (PDOException $e) {

err("Error!: ".$e->getMessage()."<br/>");

die();

}}

define("DB_USER","user");

define("DB_PASSWORD","password");

define("DB_HOST","hostname");

Page 42: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

VIII

define("DB_NAME","dbname");

dbConnect(); ?>

Page 43: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

IX

Appendix G - Database structure

The table structure in the MySQL database.

CREATE TABLE `Clicks` (

`ID` int(11) NOT NULL AUTO_INCREMENT;,

`linkID` int(11) NOT NULL,

`Clicks` int(10) NOT NULL,

`fullTerm` varchar(80) NOT NULL,

`Term` varchar(30) NOT NULL,

SET PRIMARY KEY(`ID`)

) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Page 44: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

X

Appendix H - Clickthrough Data Algorithm

The algorithm rescoring documents depending on search terms and clicks.

package org.apache.solr.search;

import java.io.BufferedReader;

import java.io.IOException;

import java.io.InputStream;

import java.io.InputStreamReader;

import java.io.Reader;

import java.net.URL;

import java.nio.charset.Charset;

import java.util.ArrayList;

import java.util.List;

import java.util.regex.Matcher;

import java.util.regex.Pattern;

import org.json.JSONException;

import org.json.JSONObject;

/**

* Clickthrough Data Algorithm

* <p> Used by the @see ClickthroughDataQParserPlugin to rescore documents.

*/

public class ClickthroughDataAlgorithm {

List<String> words = new ArrayList<String>();

private String query;

JSONObject json;

public ClickthroughDataAlgorithm (String query) throws JSONException,

IOException{

this.query = query;

this.json = readJsonFromUrl("INSERT API HERE"+getQueryString());

}

/**

* Reads the JSON object and returns it as a String.

*/

private static String readAll(Reader rd) throws IOException {

StringBuilder sb = new StringBuilder();

int cp;

while ((cp = rd.read()) != -1) {

sb.append((char) cp);

}

return sb.toString();

}

/**

* Collects the JSON object from the API

*/

Page 45: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XI

public static JSONObject readJsonFromUrl(String url) throws IOException,

JSONException {

InputStream is = new URL(url).openStream();

String jsonText = null;

try {

BufferedReader rd = new BufferedReader(new InputStreamReader(is,

Charset.forName("UTF-8")));

jsonText = readAll(rd);

JSONObject json = new JSONObject(jsonText.replaceAll("\\<.*?>",""));

return json;

} finally {

is.close();

}

}

public String getQueryString(){

String regex = "([a-zA-Z]+)";

Pattern p = Pattern.compile(regex, 2);

Matcher m = p.matcher(query);

while (m.find()) {

String word = m.group().toString();

if(!(word.equals("Title") || word.equals("Content") || word.equals("Tags")

|| word.equals("AND") || word.equals("OR"))){

if(!(words.contains(word))){

words.add(m.group());

}

}

}

StringBuilder sb = new StringBuilder();

for(String s : words){

sb.append(" ");

sb.append(s);

}

return (sb.toString().substring(1));

}

/**

* Computes a new score for the document

*/

public float score(String id) throws IOException{

String queryString = getQueryString();

float clickscore = 0;

//Compares the document id Clicks to the query.

//A fullterm match grants twice as high score as a single search term.

for(int i = 0; i < json.getJSONArray("results").length(); i++){

JSONObject result = json.getJSONArray("results").getJSONObject(i);

String fullTerm = result.getString("fullTerm");

String linkID = result.getString("linkID");

if(id.equals(linkID)){

Page 46: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XII

if(queryString.equals(fullTerm)){

clickscore += (result.getInt("Clicks") * 2);

}else{

int matchCount = 0;

float wordClickScore = 0;

for(int i2 = 0; i2 < words.size(); i2++){

String word = words.get(i2);

String term = result.getString("Term");

if(word.equals(term)){

wordClickScore += (result.getInt("Clicks"));

matchCount += 1;

}

clickscore += wordClickScore / matchCount;

}

}

}

}

return clickscore;

}

}

Page 47: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XIII

Appendix I - ClickthroughDataQParserPlugin

The Clickthrough Data Query Parser Plugin source code.

package org.apache.solr.search;

import java.io.IOException;

import java.util.Arrays;

import java.util.Comparator;

import java.util.List;

import org.apache.lucene.index.LeafReaderContext;

import org.apache.lucene.search.Explanation;

import org.apache.lucene.search.IndexSearcher;

import org.apache.lucene.search.MatchAllDocsQuery;

import org.apache.lucene.search.Query;

import org.apache.lucene.search.Rescorer;

import org.apache.lucene.search.ScoreDoc;

import org.apache.lucene.search.Scorer;

import org.apache.lucene.search.TopDocs;

import org.apache.lucene.search.Weight;

import org.apache.solr.common.SolrException;

import org.apache.solr.common.params.CommonParams;

import org.apache.solr.common.params.SolrParams;

import org.apache.solr.request.SolrQueryRequest;

/*

* Optional Params:

* cdaDocs - Defaults at top 200 docs

* cdaWeight - Defaults at 1.0

*

* Syntax: q=*:*&rq={!cda cdaDocs=300 cdaWeight=3}

*

*/

/**

* Query parser for the Clickthrough Data Algorithm addition

*/

public class ClickthroughDataQParserPlugin extends QParserPlugin{

public static final String NAME = "cda";

private static Query defaultQuery = new MatchAllDocsQuery();

public static final String CDA_WEIGHT = "cdaWeight";

public static final double CDA_WEIGHT_DEFAULT = 1.0d;

public static final String CDA_DOCS = "cdaDocs";

public static final int CDA_DOCS_DEFAULT = 200;

public QParser createParser(String query, SolrParams localParams, SolrParams

params, SolrQueryRequest req) {

return new cdaQParser(query, localParams, params, req);

}

Page 48: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XIV

private class cdaQParser extends QParser {

public cdaQParser(String query, SolrParams localParams, SolrParams params,

SolrQueryRequest req) {

super(query, localParams, params, req);

}

public Query parse() throws SyntaxError {

String quer = params.get(CommonParams.Q);

if(quer == null || quer.trim().length() == 0){

throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Query cannot

be null");

}

QParser qp = QParser.getParser(quer, req);

Query qry = qp.getQuery();

if(qry.toString() == null){

throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Querystring

cannot be null");

}

int docCount = localParams.getInt(CDA_DOCS, CDA_DOCS_DEFAULT);

double weight = localParams.getDouble(CDA_WEIGHT, CDA_WEIGHT_DEFAULT);

return new ClickthroughDataQuery(qry, docCount, weight);

}

}

private final class ClickthroughDataQueryRescorer extends Rescorer {

private final double reRankWeight;

private final Query query;

private final float scoreSize = 0.1f;

public ClickthroughDataQueryRescorer(Query reRankQuery, double reRankWeight) {

this.query = reRankQuery;

this.reRankWeight = reRankWeight;

}

protected float combine(float initialScore, boolean secondPassMatches, float

newScore) {

float score = initialScore;

if (secondPassMatches) {

score += reRankWeight * (newScore * scoreSize);

}

return score;

}

@Override

Page 49: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XV

public TopDocs rescore(IndexSearcher searcher, TopDocs firstPassTopDocs, int

topN) throws IOException {

ScoreDoc[] hits = firstPassTopDocs.scoreDocs.clone();

Arrays.sort(hits,

new Comparator<ScoreDoc>() {

@Override

public int compare(ScoreDoc a, ScoreDoc b) {

return a.doc - b.doc;

}

});

List<LeafReaderContext> leaves = searcher.getIndexReader().leaves();

Weight weight = searcher.createNormalizedWeight(query, true);

int hitUpto = 0;

int readerUpto = -1;

int endDoc = 0;

int docBase = 0;

Scorer scorer = null;

ClickthroughDataAlgorithm cda = new

ClickthroughDataAlgorithm(query.toString());

while (hitUpto < hits.length) {

ScoreDoc hit = hits[hitUpto];

int docID = hit.doc;

LeafReaderContext readerContext = null;

while (docID >= endDoc) {

readerUpto++;

readerContext = leaves.get(readerUpto);

endDoc = readerContext.docBase + readerContext.reader().maxDoc();

}

if (readerContext != null) {

docBase = readerContext.docBase;

scorer = weight.scorer(readerContext);

}

if(scorer != null) {

int targetDoc = docID - docBase;

int actualDoc = scorer.docID();

if (actualDoc < targetDoc) {

actualDoc = scorer.iterator().advance(targetDoc);

}

if (actualDoc == targetDoc) {

// Query did match this doc

hit.score = combine(hit.score, true,

cda.score(searcher.doc(actualDoc).get("Id")));

} else {

// Query did not match this doc

assert actualDoc > targetDoc;

Page 50: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XVI

hit.score = combine(hit.score, false, 0.0f);

}

} else {

// Query did not match this doc

hit.score = combine(hit.score, false, 0.0f);

}

hitUpto++;

}

Arrays.sort(hits,

new Comparator<ScoreDoc>() {

@Override

public int compare(ScoreDoc a, ScoreDoc b) {

if (a.score > b.score) {

return -1;

} else if (a.score < b.score) {

return 1;

} else {

return a.doc - b.doc;

}

}

});

if (topN < hits.length) {

ScoreDoc[] subset = new ScoreDoc[topN];

System.arraycopy(hits, 0, subset, 0, topN);

hits = subset;

}

return new TopDocs(firstPassTopDocs.totalHits, hits, hits[0].score);

}

@Override

public Explanation explain(IndexSearcher searcher, Explanation

firstPassExplanation, int docID) throws IOException {

return null;

}

}

private final class ClickthroughDataQuery extends AbstractReRankQuery {

final private Query ClickthroughDataQuery;

final private double reRankWeight;

public ClickthroughDataQuery(Query Query, int docCount, double weight) {

super(defaultQuery, docCount, new ClickthroughDataQueryRescorer(Query,

weight));

this.reRankWeight = weight;

this.ClickthroughDataQuery = Query;

}

Page 51: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XVII

public int hashCode() {

return 31 * classHash() +

mainQuery.hashCode()+ClickthroughDataQuery.hashCode()+(int)reRankWeight+reRankDocs

;

}

public boolean equals(Object other) {

return sameClassAs(other) &&

equalsTo(getClass().cast(other));

}

private boolean equalsTo(ClickthroughDataQuery cdaq) {

return mainQuery.equals(cdaq.mainQuery) &&

ClickthroughDataQuery.equals(cdaq.ClickthroughDataQuery) &&

reRankWeight == cdaq.reRankWeight &&

reRankDocs == cdaq.reRankDocs;

}

@Override

public String toString(String s) {

final StringBuilder sb = new StringBuilder(100);

sb.append("{!").append(NAME);

sb.append(" Query='").append(mainQuery.toString()).append("' ");

sb.append(CDA_DOCS).append('=').append(reRankDocs).append(' ');

sb.append(CDA_WEIGHT).append('=').append(reRankWeight).append('}');

return sb.toString();

}

protected Query rewrite(Query rewrittenMainQuery) throws IOException {

return new ClickthroughDataQuery(ClickthroughDataQuery, reRankDocs,

reRankWeight).wrap(rewrittenMainQuery);

}

}

}

Page 52: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XVIII

Appendix J - Search Platform PHP

The php section of the search platform.

<html>

<head>

<!-- Bootstrap -->

<link rel="stylesheet"

href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">

<link rel="stylesheet"

href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap-

theme.min.css">

<script

src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>

<script

src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script

>

<!-- Bootstrap -->

</head>

<body>

<div class="container" style="">

<div class="container">

<h3>Search</h3>

<form class="form-inline" action="./" method="get">

<input id="search" <?php echo (isset($_GET[ 'q']) ? 'value="'.

$_GET[ 'q'] . '"': 'placeholder="Search..."'); ?> class="form-control"

style="width:85%;" type="text" name="q"/>

<button type="submit" class="btn btn-default"><span

class="glyphicon glyphicon-search" aria-hidden="true"></span> Search</button>

</form>

</div>

</div>

<div class="container">

<?php

if(isset($_GET['q'])){

Page 53: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XIX

//Building the search query -- ex, "install

windows" turns in to: "(Title:install AND Title:windows) OR (Content:install AND

Content:windows) OR (Tags:install AND Tags:windows)".

$qw = explode(' ',strtolower($_GET['q']));

$qTitle = implode('%20AND%20Title:',$qw);

$qContent = implode('%20AND%20Content:',$qw);

$qTags = implode('%20OR%20Tags:',$qw);

$query = '(Title:'. $qTitle.')%20OR%20(Content:'.

$qContent .')%20OR%20(Tags:'. $qTags .')';

//Gets the number of results

$getResultCount =

'http://83.252.83.212:8999/solr/exam/select?fl=Id&fq=PostType:1&indent=on&q='.

$query .'&wt=json&rows=1';

$countJson = file_get_contents($getResultCount);

$numFound = json_decode($countJson);

$total = $numFound->{'response'}->{'numFound'};

//Limit of search results per page

$limit = 10;

$pages = ceil($total / $limit);

//Current Page

$page = min($pages, filter_input(INPUT_GET,

'page', FILTER_VALIDATE_INT, array(

'options' => array(

'default' => 1,

'min_range' => 1,

),

)));

Page 54: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XX

$offset = ($page - 1) * $limit;

$start = $offset + 1;

$end = min(($offset + $limit), $total);

//checks if there are any results and then prints

the search results if there is any.

if($total > 0){

//Timer to see how long the search was

$startTime = microtime(true);

//Performs the actual search -- Might be

able to use same as number of results -- look at this

$searchString =

'http://83.252.83.212:8999/solr/exam/select?fl=Id,%20Title,%20Content,%20score&fq=

PostType:1&indent=on&q='. $query .'&wt=json&rows='. $limit .'&start='. $start;

$json = file_get_contents($searchString);

$parsedJson = json_decode($json);

echo "<div class='result col-md-6'>";

echo "<p> Found ". $parsedJson-

>{'response'}->{'numFound'} ." results in <span id='info'></span></p>";

foreach($parsedJson->{'response'}-

>{'docs'} as $item){

$content = strip_tags($item-

>{'Content'});

if(strlen($content) > 50){

$content = substr($content,

0, 160) . "...";

}

echo "<a id='". $item->{'Id'} ."'

class='docTitle' href='http://www.stackoverflow.com/questions/". $item->{'Id'} ."'

target='_blank'>". htmlspecialchars(ucfirst($item->{'Title'})) ."</a>";

echo "<p>". $content ."</p>";

Page 55: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXI

}

echo "</div>";

//Timer to see how long the search was

$endTime = microtime(true);

//Timer to see how long the search was

$cdaStartTime = microtime(true);

//Performs the clickthrough data algorithm

search -- Might be able to use same as number of results -- look at this

$cdaSearchString =

'http://83.252.83.212:8999/solr/exam/select?fl=Id,%20Title,%20Content,%20score&fq=

PostType:1&indent=on&q='. $query .'&rq={!cda%20cdaWeight=5}&wt=json&rows='. $limit

.'&start='. $start;

$cdaJson =

file_get_contents($cdaSearchString);

$parsedCdaJson = json_decode($cdaJson);

echo "<div class='result col-md-6'>";

echo "<p> Found ". $parsedCdaJson-

>{'response'}->{'numFound'} ." results in <span id='cdainfo'></span></p>";

foreach($parsedCdaJson->{'response'}-

>{'docs'} as $cdaItem){

$cdaContent = strip_tags($cdaItem-

>{'Content'});

if(strlen($cdaContent) > 50){

$cdaContent =

substr($cdaContent, 0, 160) . "...";

}

echo "<a id='". $cdaItem->{'Id'} ."'

class='docTitle' href='http://www.stackoverflow.com/questions/". $cdaItem->{'Id'}

."' target='_blank'>". htmlspecialchars(ucfirst($cdaItem->{'Title'})) ."</a>";

echo "<p>". $cdaContent ."</p>";

Page 56: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXII

}

echo "</div>";

?>

<div class='col-md-12'>

<nav aria-label="Page navigation">

<?php echo '<p style="margin:20px 0 -15px 0;">Page ', $page, '

of ', $pages, ' pages, displaying result ', $start, '-', $end, ' of ', $total, '

results.</p>'; ?>

<ul class="pagination">

<li>

<a href="<?php echo '?page=' . ($page - 1) . '&q='.

$_GET['q']; ?>" aria-label="Previous">

<span aria-hidden="true">&laquo;</span>

</a>

</li>

<li>

<a href="<?php echo '?page=' . ($page + 1) . '&q='.

$_GET['q']; ?>" aria-label="Next">

<span aria-hidden="true">&raquo;</span>

</a>

</li>

</ul>

</nav>

</div>

<?php

}else{

echo "No results found";

}

//Timer to see how long the search was

$cdaEndTime = microtime(true);

}?>

Page 57: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXIII

</div>

<!-- Prints the time it took for the search to be performed. -->

<script>

var time = "<?php $loadTime = $endTime - $startTime; echo

(round($loadTime,2)!= 1 ? round($loadTime,2) .' seconds.' : round($loadTime,2) .'

second.'); ?>";

var cdatime = "<?php $cdaLoadTime = $cdaEndTime - $cdaStartTime; echo

(round($cdaLoadTime,2)!= 1 ? round($cdaLoadTime,2) .' seconds.' :

round($cdaLoadTime,2) .' second.'); ?>";

$(window).load(function(e) {

$("#info").html(time);

$("#cdainfo").html(cdatime);

});

</script>

<!-- import JS document -->

<script src="js/main.js"></script>

</body>

</html>

Page 58: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXIV

Appendix K - JavaScript

The JavaScript code to manage clicks and spam prevention.

$(function(){

$(".docTitle").click(function(){

updateClick($(this).attr('id'));

});

});

function updateClick(id){

var fullTerm = $('#search').val();

if(getCookie(fullTerm.replace(" ", "+") + id) == null){

$.ajax({

type: 'POST',

url: 'functions/click.php',

data: {

'ID' : id,

'fullTerm' : fullTerm,

}

});

var date = new Date();

date.setTime(date.getTime() + (60 * 1000));

setCookie(fullTerm.replace(" ", "+") + id, id, { expires: date });

}

}

function setCookie(name, value, minutes) {

var expires = "";

if (minutes) {

var date = new Date();

date.setTime(date.getTime() + (minutes * 60 * 1000));

expires = "; expires=" + date.toUTCString();

Page 59: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXV

}

document.cookie = name + "=" + value + expires + "; path=/";

}

function getCookie(name) {

var nameEQ = name + "=";

var ca = document.cookie.split(';');

for(var i=0;i < ca.length;i++) {

var c = ca[i];

while (c.charAt(0)==' ') c = c.substring(1,c.length);

if (c.indexOf(nameEQ) == 0){

return c.substring(nameEQ.length,c.length);

}

}

return null;

}

Page 60: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXVI

Appendix L - Pilot Study Time Measurement

The full list of measurments of the pilot study.

Row CDA activated? true/false Query Search request time

1 0 windows install 0.39066696166992

2 1 windows install 0.40063095092773

3 0 windows install 0.38576793670654

4 1 windows install 0.39710307121277

5 0 windows install 0.39480805397034

6 1 windows install 0.38832211494446

7 0 windows install 0.19270706176758

8 1 windows install 0.18550896644592

9 0 windows install 0.39270114898682

10 1 windows install 0.19290208816528

11 0 windows install 0.39673590660095

12 1 windows install 0.41673898696899

13 0 PHP Shorthand Usage 0.24891495704651

14 1 PHP Shorthand Usage 0.52767491340637

15 0 JavaScript Cookies 0.38677191734314

16 1 JavaScript Cookies 0.5347797870636

17 0 Apache Solr Configuration 0.17385697364807

18 1 Apache Solr Configuration 0.31360483169556

19 0 SHA256 vs SHA512 0.26769304275513

20 1 SHA256 vs SHA512 0.55362796783447

21 0 Export PDF 0.37069988250732

22 1 Export PDF 0.52175903320312

23 0 Virtualbox Linux Guide 0.20557284355164

24 1 Virtualbox Linux Guide 0.34202194213867

25 0 Virtualbox Linux Guide 0.37069988250732

26 1 Virtualbox Linux Guide 0.52175903320312

27 0 Export PDF 0.26769304275513

28 1 Export PDF 0.55362796783447

29 0 SHA256 vs SHA512 0.17385697364807

30 1 SHA256 vs SHA512 0.31360483169556

31 0 Apache Solr Configuration 0.38677191734314

32 1 Apache Solr Configuration 0.5347797870636

33 0 SHA256 vs SHA512 0.17385697364807

34 1 SHA256 vs SHA512 0.31360483169556

35 0 Export PDF 0.26769304275513

36 1 Export PDF 0.55362796783447

37 0 SHA256 vs SHA512 0.17385697364807

38 1 SHA256 vs SHA512 0.31360483169556

39 0 Apache Solr Configuration 0.38677191734314

40 1 Apache Solr Configuration 0.5347797870636

41 0 JavaScript Cookies 0.24891495704651

42 1 JavaScript Cookies 0.52767491340637

43 0 JavaScript Cookies 0.24891495704651

Page 61: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXVII

44 1 JavaScript Cookies 0.52767491340637

45 0 Apache Solr Configuration 0.38677191734314

46 1 Apache Solr Configuration 0.5347797870636

47 0 JavaScript Cookies 0.24891495704651

48 1 JavaScript Cookies 0.52767491340637

49 0 Apache Solr Configuration 0.38677191734314

50 1 Apache Solr Configuration 0.5347797870636

51 0 SHA256 vs SHA512 0.17385697364807

52 1 SHA256 vs SHA512 0.31360483169556

53 0 Export PDF 0.26769304275513

54 1 Export PDF 0.55362796783447

55 0 SHA256 vs SHA512 0.17385697364807

56 1 SHA256 vs SHA512 0.31360483169556

57 0 Export PDF 0.26769304275513

58 1 Export PDF 0.55362796783447

59 0 Virtualbox Linux Guide 0.37069988250732

60 1 Virtualbox Linux Guide 0.52175903320312

61 0 Export PDF 0.26769304275513

62 1 Export PDF 0.55362796783447

63 0 Virtualbox Linux Guide 0.37069988250732

64 1 Virtualbox Linux Guide 0.52175903320312

65 0 Export PDF 0.15653491020203

66 1 Export PDF 0.36192297935486

67 0 Virtualbox Linux Guide 0.20557284355164

68 1 Virtualbox Linux Guide 0.34202194213867

69 0 Export PDF 0.15653491020203

70 1 Export PDF 0.36192297935486

71 0 Virtualbox Linux Guide 0.20557284355164

72 1 Virtualbox Linux Guide 0.34202194213867

73 0 Export PDF 0.15653491020203

74 1 Export PDF 0.36192297935486

75 0 Virtualbox Linux Guide 0.20557284355164

76 1 Virtualbox Linux Guide 0.34202194213867

77 0 Virtualbox Linux Guide 0.20202803611755

78 1 Virtualbox Linux Guide 0.23084688186646

79 0 Export PDF 0.15653491020203

80 1 Export PDF 0.36192297935486

81 0 Export PDF 0.26769304275513

82 1 Export PDF 0.55362796783447

83 0 SHA256 vs SHA512 0.46341300010681

84 1 SHA256 vs SHA512 0.47106289863586

85 0 SHA256 vs SHA512 0.17385697364807

86 1 SHA256 vs SHA512 0.31360483169556

87 0 Apache Solr Configuration 0.35405707359314

88 1 Apache Solr Configuration 0.15385699272156

89 0 Apache Solr Configuration 0.38677191734314

90 1 Apache Solr Configuration 0.5347797870636

91 0 JavaScript Cookies 0.24891495704651

Page 62: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXVIII

92 1 JavaScript Cookies 0.52767491340637

93 0 PHP Shorthand Usage 0.23060917854309

94 1 PHP Shorthand Usage 0.23978304862976

95 0 PHP Shorthand Usage 0.23414993286133

96 1 PHP Shorthand Usage 0.23700499534607

97 0 PHP Shorthand Usage 0.43578386306763

98 1 PHP Shorthand Usage 0.2449209690094

99 0 Apache Solr Configuration 0.35354900360107

100 1 Apache Solr Configuration 0.34358692169189

101 0 Apache Solr Configuration 0.34776902198792

102 1 Apache Solr Configuration 0.35608911514282

103 0 Apache Solr Configuration 0.35722494125366

104 1 Apache Solr Configuration 0.35953307151794

105 0 Apache Solr Configuration 0.15553593635559

106 1 Apache Solr Configuration 0.35929298400879

107 0 Apache Solr Configuration 0.3512179851532

108 1 Apache Solr Configuration 0.1509850025177

109 0 Apache Solr Configuration 0.35394692420959

110 1 Apache Solr Configuration 0.15144300460815

111 0 Apache Solr Configuration 0.15291404724121

112 1 Apache Solr Configuration 0.15262413024902

113 0 Apache Solr Configuration 0.35674405097961

114 1 Apache Solr Configuration 0.35676217079163

115 0 Apache Solr Configuration 0.15112709999084

116 1 Apache Solr Configuration 0.15124297142029

117 0 Apache Solr Configuration 0.15461397171021

118 1 Apache Solr Configuration 0.15570998191833

119 0 Apache Solr Configuration 0.15599489212036

120 1 Apache Solr Configuration 0.35425901412964

121 0 Apache Solr Configuration 0.35553312301636

122 1 Apache Solr Configuration 0.15792298316956

123 0 Apache Solr Configuration 0.35507583618164

124 1 Apache Solr Configuration 0.35042095184326

125 0 Export PDF 0.14722013473511

126 1 Export PDF 0.34592390060425

127 0 Export PDF 0.35133385658264

128 1 Export PDF 0.35535883903503

129 0 Export PDF 0.35546708106995

130 1 Export PDF 0.35618710517883

131 0 Export PDF 0.15665292739868

132 1 Export PDF 0.36356902122498

133 0 Export PDF 0.15014290809631

134 1 Export PDF 0.15329885482788

135 0 Export PDF 0.1516478061676

136 1 Export PDF 0.35736203193665

137 0 Export PDF 0.3490309715271

138 1 Export PDF 0.15919995307922

139 0 Export PDF 0.34838104248047

Page 63: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXIX

140 1 Export PDF 0.35613107681274

141 0 Export PDF 0.16556000709534

142 1 Export PDF 0.35575604438782

143 0 JavaScript Cookies 0.35084509849548

144 1 JavaScript Cookies 0.15147399902344

145 0 JavaScript Cookies 0.35812497138977

146 1 JavaScript Cookies 0.14855885505676

147 0 JavaScript Cookies 0.35270690917969

148 1 JavaScript Cookies 0.35968613624573

149 0 JavaScript Cookies 0.3557460308075

150 1 JavaScript Cookies 0.35600709915161

151 0 JavaScript Cookies 0.35389018058777

152 1 JavaScript Cookies 0.34264707565308

153 0 JavaScript Cookies 0.35396003723145

154 1 JavaScript Cookies 0.15394997596741

155 0 JavaScript Cookies 0.14723801612854

156 1 JavaScript Cookies 0.34529399871826

157 0 JavaScript Cookies 0.35309100151062

158 1 JavaScript Cookies 0.15855884552002

159 0 JavaScript Cookies 0.35502791404724

160 1 JavaScript Cookies 0.35020899772644

161 0 JavaScript Cookies 0.15011620521545

162 1 JavaScript Cookies 0.34997487068176

163 0 JavaScript Cookies 0.15071415901184

164 1 JavaScript Cookies 0.15184497833252

165 0 JavaScript Cookies 0.35467886924744

166 1 JavaScript Cookies 0.1537458896637

167 0 JavaScript Cookies 0.35624885559082

168 1 JavaScript Cookies 0.16572093963623

169 0 JavaScript Cookies 0.15172410011292

170 1 JavaScript Cookies 0.35715699195862

171 0 JavaScript Cookies 0.15052890777588

172 1 JavaScript Cookies 0.3521728515625

173 0 PHP Shorthand Usage 0.44709706306458

174 1 PHP Shorthand Usage 0.42663216590881

175 0 PHP Shorthand Usage 0.43002796173096

176 1 PHP Shorthand Usage 0.23661017417908

177 0 PHP Shorthand Usage 0.44004607200623

178 1 PHP Shorthand Usage 0.43934011459351

179 0 PHP Shorthand Usage 0.22632098197937

180 1 PHP Shorthand Usage 0.22606801986694

181 0 PHP Shorthand Usage 0.22881698608398

182 1 PHP Shorthand Usage 0.23870587348938

183 0 PHP Shorthand Usage 0.22734498977661

184 1 PHP Shorthand Usage 0.23638486862183

185 0 PHP Shorthand Usage 0.43230295181274

186 1 PHP Shorthand Usage 0.43383479118347

187 0 PHP Shorthand Usage 0.2426609992981

Page 64: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXX

188 1 PHP Shorthand Usage 0.24085307121277

189 0 PHP Shorthand Usage 0.4343249797821

190 1 PHP Shorthand Usage 0.44498491287231

191 0 PHP Shorthand Usage 0.2320990562439

192 1 PHP Shorthand Usage 0.42316007614136

193 0 PHP Shorthand Usage 0.22956895828247

194 1 PHP Shorthand Usage 0.43605494499207

195 0 PHP Shorthand Usage 0.22338199615479

196 1 PHP Shorthand Usage 0.23057103157043

197 0 PHP Shorthand Usage 0.43777084350586

198 1 PHP Shorthand Usage 0.4348840713501

199 0 PHP Shorthand Usage 0.43612790107727

200 1 PHP Shorthand Usage 0.43511199951172

201 0 PHP Shorthand Usage 0.4296600818634

202 1 PHP Shorthand Usage 0.43144083023071

203 0 PHP Shorthand Usage 0.43994903564453

204 1 PHP Shorthand Usage 0.23462700843811

205 0 SHA256 vs SHA512 0.46828889846802

206 1 SHA256 vs SHA512 0.46612882614136

207 0 SHA256 vs SHA512 0.26188492774963

208 1 SHA256 vs SHA512 0.45900297164917

209 0 SHA256 vs SHA512 0.46511888504028

210 1 SHA256 vs SHA512 0.47402000427246

211 0 SHA256 vs SHA512 0.26856994628906

212 1 SHA256 vs SHA512 0.47076082229614

213 0 SHA256 vs SHA512 0.2756040096283

214 1 SHA256 vs SHA512 0.47590184211731

215 0 SHA256 vs SHA512 0.27258205413818

216 1 SHA256 vs SHA512 0.27507400512695

217 0 SHA256 vs SHA512 0.4624879360199

218 1 SHA256 vs SHA512 0.26160907745361

219 0 SHA256 vs SHA512 0.46631383895874

220 1 SHA256 vs SHA512 0.46281099319458

221 0 SHA256 vs SHA512 0.46294808387756

222 1 SHA256 vs SHA512 0.46066498756409

223 0 SHA256 vs SHA512 0.27313899993896

224 1 SHA256 vs SHA512 0.27754521369934

225 0 SHA256 vs SHA512 0.28166198730469

226 1 SHA256 vs SHA512 0.27400183677673

227 0 SHA256 vs SHA512 0.47555208206177

228 1 SHA256 vs SHA512 0.2743079662323

229 0 Virtualbox Linux Guide 0.3899199962616

230 1 Virtualbox Linux Guide 0.42860007286072

231 0 Virtualbox Linux Guide 0.19223594665527

232 1 Virtualbox Linux Guide 0.22263789176941

233 0 Virtualbox Linux Guide 0.38838696479797

234 1 Virtualbox Linux Guide 0.43244695663452

235 0 Virtualbox Linux Guide 0.38974905014038

Page 65: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXI

236 1 Virtualbox Linux Guide 0.23240900039673

237 0 Virtualbox Linux Guide 0.38727307319641

238 1 Virtualbox Linux Guide 0.23549389839172

239 0 Virtualbox Linux Guide 0.21151399612427

240 1 Virtualbox Linux Guide 0.43374300003052

241 0 Virtualbox Linux Guide 0.19044709205627

242 1 Virtualbox Linux Guide 0.23159098625183

243 0 Virtualbox Linux Guide 0.19573402404785

244 1 Virtualbox Linux Guide 0.23290705680847

245 0 Virtualbox Linux Guide 0.39542484283447

246 1 Virtualbox Linux Guide 0.22225594520569

247 0 Virtualbox Linux Guide 0.19355201721191

248 1 Virtualbox Linux Guide 0.42516899108887

249 0 Virtualbox Linux Guide 0.19699215888977

250 1 Virtualbox Linux Guide 0.23267102241516

251 0 Virtualbox Linux Guide 0.39587092399597

252 1 Virtualbox Linux Guide 0.43860006332397

253 0 windows install 0.19268488883972

254 1 windows install 0.19681715965271

255 0 windows install 0.38778591156006

256 1 windows install 0.39219498634338

257 0 windows install 0.37761497497559

258 1 windows install 0.38985395431519

259 0 windows install 0.18655681610107

260 1 windows install 0.39090085029602

261 0 windows install 0.38849806785583

262 1 windows install 0.39074492454529

263 0 windows install 0.19604802131653

264 1 windows install 0.19575500488281

265 0 windows install 0.39476490020752

266 1 windows install 0.40371084213257

267 0 windows install 0.19091606140137

268 1 windows install 0.38266205787659

269 0 windows install 0.18472695350647

270 1 windows install 0.18455505371094

271 0 windows install 0.39416694641113

272 1 windows install 0.19606518745422

273 0 windows install 0.39487910270691

274 1 windows install 0.40010595321655

275 0 windows install 0.39213490486145

276 1 windows install 0.19361400604248

277 0 windows install 0.38638401031494

278 1 windows install 0.19212102890015

279 0 windows install 0.38638401031494

280 1 windows install 0.19212102890015

Page 66: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXII

Appendix M - Clickthrough Data Algorithm

Measurement Results

Clickthrough Data Algorithm measurement results for the efficiency evaluation.

Index sizes: 1 = 5000 documents, 2 = 50 000 documents, 3 = 1 000 000 documents

Query Number Index Size Query Solr Response Time (s) Site Load Time (s) CDA Run Time (s)

1 1 windows install 0.3932020664 0.9180000000 0.0025681325

2 1 windows install 0.3952789307 0.9130000000 0.0013217926

3 1 windows install 0.1966941357 0.4910000000 0.1092998981

4 1 windows install 0.3906109333 0.9430000000 0.0004811287

5 1 windows install 0.4009418488 0.9160000000 0.0001509190

6 1 windows install 0.1888148785 0.7140000000 0.0017406940

7 1 windows install 0.3888220787 0.9220000000 0.0012488365

8 1 windows install 0.3924331665 0.7370000000 0.0006618500

9 1 windows install 0.3967719078 0.7060000000 0.0007910728

10 1 windows install 0.3864910603 0.9430000000 0.0019400120

11 1 windows install 0.3928339481 0.9160000000 0.0009810925

12 1 windows install 0.3906750679 0.9260000000 0.0004990101

13 1 windows install 0.3904538155 0.9450000000 0.0003561974

14 1 windows install 0.1929459572 0.7010000000 0.0038681030

15 1 windows install 0.1931669712 0.7370000000 0.0042860508

16 1 windows install 0.3984401226 0.9540000000 0.0004029274

17 1 windows install 0.3905720711 0.9540000000 0.0004169941

18 1 windows install 0.3885068893 0.9150000000 0.0006988049

19 1 windows install 0.3825879097 0.8910000000 0.0032861233

20 1 windows install 0.3887560368 0.9180000000 0.0011491776

21 1 windows install 0.3952419758 0.9510000000 0.0013599396

22 1 windows install 0.3905138969 0.9210000000 0.0003857613

23 1 windows install 0.3967540264 0.9250000000 0.0008041859

24 1 windows install 0.3867540359 0.9300000000 0.0017790794

25 1 windows install 0.1925911903 0.7050000000 0.0033929348

26 1 windows install 0.3868689537 0.7160000000 0.0017328262

27 1 windows install 0.3805429935 0.9270000000 0.0024166107

28 1 windows install 0.3966438770 0.9480000000 0.0005779266

29 1 windows install 0.1883430481 0.7380000000 0.0010578632

30 1 windows install 0.1887099743 0.6970000000 0.0016837120

31 1 windows install 0.4008300304 0.9490000000 0.0000109673

32 1 windows install 0.3909900188 0.6940000000 0.0011570454

33 1 windows install 0.1885459423 0.7210000000 0.0017261505

34 1 windows install 0.3867011070 0.9050000000 0.0019018650

35 1 windows install 0.1908149719 0.7240000000 0.0017321110

36 1 windows install 0.3807790279 0.8870000000 0.0028247833

37 1 windows install 0.1923830509 0.7300000000 0.0023579597

38 1 windows install 0.3905479908 0.9240000000 0.0003840923

39 1 windows install 0.3926920891 0.9330000000 0.0006639957

40 1 windows install 0.3945949078 0.9260000000 0.0006930828

Page 67: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXIII

41 1 windows install 0.1847019196 0.7010000000 0.0017800331

42 1 windows install 0.4006910324 0.9650000000 0.0000300407

43 1 windows install 0.3926370144 0.7390000000 0.0006210804

44 1 windows install 0.1869330406 0.5180000000 0.0015900135

45 1 windows install 0.3782830238 0.9340000000 0.0041952133

46 1 windows install 0.3906590939 0.9220000000 0.0004799366

47 1 windows install 0.3876681328 0.9220000000 0.0012359619

48 1 windows install 0.3919880390 0.9140000000 0.0008080006

49 1 windows install 0.3982789516 0.9350000000 0.0005018711

50 1 windows install 0.1866269112 0.7380000000 0.0006470680

51 1 windows install 0.3901441097 0.9100000000 0.0004968643

52 1 windows install 0.3784868717 0.9380000000 0.0037212372

53 1 windows install 0.3864998817 0.9160000000 0.0019431114

54 1 windows install 0.3851349354 0.9580000000 0.0023248196

55 1 windows install 0.3889219761 0.7360000000 0.0012412071

56 1 windows install 0.3915150166 0.9080000000 0.0011219978

57 1 windows install 0.1967210770 0.7000000000 0.1092729568

58 1 windows install 0.3940699100 0.9140000000 0.0008361340

59 1 windows install 0.3910570145 0.9520000000 0.0011940002

60 1 windows install 0.3847100735 0.9270000000 0.0025591850

61 1 windows install 0.4003739357 0.9150000000 0.0000562668

62 1 windows install 0.1885859966 0.7270000000 0.0017168522

63 1 windows install 0.3914461136 0.9540000000 0.0011439323

64 1 windows install 0.3904898167 0.9240000000 0.0003831387

65 1 windows install 0.3925950527 0.9300000000 0.0006189346

66 1 windows install 0.3901798725 0.8830000000 0.0004999638

67 1 windows install 0.3927819729 0.9670000000 0.0008149147

68 1 windows install 0.3866031170 0.9100000000 0.0019500256

69 1 windows install 0.1895070076 0.7070000000 0.0012092590

70 1 windows install 0.3946630955 0.8930000000 0.0007662773

71 1 windows install 0.4033730030 0.9580000000 0.0003821850

72 1 windows install 0.3837058544 0.7430000000 0.0027537346

73 1 windows install 0.3823890686 0.9060000000 0.0024912357

74 1 windows install 0.1856598854 0.7120000000 0.0009059906

75 1 windows install 0.3824419975 0.6890000000 0.0024969578

76 1 windows install 0.1821980476 0.7040000000 0.0019869804

77 1 windows install 0.3909230232 0.7230000000 0.0009930134

78 1 windows install 0.4042959213 0.9110000000 0.0007250309

79 1 windows install 0.1889030933 0.7250000000 0.0016679764

80 1 windows install 0.3928749561 0.9620000000 0.0011889935

81 1 windows install 0.3932700157 0.7090000000 0.0013999939

82 1 windows install 0.3886330128 0.9280000000 0.0008301735

83 1 windows install 0.3872029781 0.8960000000 0.0015680790

84 1 windows install 0.3929429054 0.9140000000 0.0013897419

85 1 windows install 0.3845839500 0.9130000000 0.0024678707

86 1 windows install 0.3885221481 0.9060000000 0.0007390976

87 1 windows install 0.3926298618 0.7390000000 0.0006299019

88 1 windows install 0.3945350647 0.7120000000 0.0006399155

Page 68: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXIV

89 1 windows install 0.3884589672 0.9050000000 0.0006930828

90 1 windows install 0.3907868862 0.9120000000 0.0007100105

91 1 windows install 0.1878199577 0.7140000000 0.0010130405

92 1 windows install 0.1912970543 0.7460000000 0.0017509460

93 1 windows install 0.1886758804 0.7150000000 0.0017077923

94 1 windows install 0.3934888840 0.9210000000 0.0012869835

95 1 windows install 0.3888361454 0.9200000000 0.0012338161

96 1 windows install 0.3906991482 0.9470000000 0.0005488396

97 1 windows install 0.3948030472 0.9200000000 0.0013561249

98 1 windows install 0.3926370144 0.7270000000 0.0006189346

99 1 windows install 0.3908679485 0.9070000000 0.0008029938

100 1 windows install 0.3967788219 0.9370000000 0.0008261204

101 1 windows install 0.3985180855 0.9570000000 0.0004191399

102 1 windows install 0.3888449669 0.9170000000 0.0012381077

103 1 windows install 0.3886818886 0.9070000000 0.0009510517

104 1 windows install 0.1867151260 0.7530000000 0.0011711121

105 1 windows install 0.3807809353 0.7110000000 0.0028381348

106 1 windows install 0.1901431084 0.7010000000 0.0009758472

107 1 windows install 0.3966600895 0.9860000000 0.0006661415

108 1 windows install 0.3883709908 0.9470000000 0.0006470680

109 1 windows install 0.3997230530 0.9540000000 0.0004692078

110 1 windows install 0.3884520531 0.9290000000 0.0006718636

111 1 windows install 0.3918211460 0.7260000000 0.0009119511

112 1 windows install 0.3934111595 0.9260000000 0.0013000965

113 1 windows install 0.4014930725 0.9160000000 0.0008878708

114 1 windows install 0.3808410168 0.9130000000 0.0026340485

115 1 windows install 0.3822209835 0.8980000000 0.0023369789

116 1 windows install 0.3885350227 0.7080000000 0.0007162094

117 1 windows install 0.1887450218 0.6980000000 0.0017442703

118 1 windows install 0.3968329430 0.7160000000 0.0007770061

119 1 windows install 0.3928070068 0.9160000000 0.0009140968

120 1 windows install 0.1876242161 0.7000000000 0.0012321472

121 1 windows install 0.3910770416 0.9270000000 0.0011961460

122 1 windows install 0.3930079937 0.9590000000 0.0014181137

123 1 windows install 0.1904041767 0.7240000000 0.0010418892

124 1 windows install 0.3899259567 0.9100000000 0.0006639957

125 1 windows install 0.3818659782 0.9280000000 0.0021760464

126 1 windows install 0.3879559040 0.9460000000 0.0008919239

127 1 windows install 0.3929669857 0.7380000000 0.0014562607

128 1 windows install 0.3924582005 0.9480000000 0.0006148815

129 1 windows install 0.3900289536 0.7130000000 0.0005109310

130 1 windows install 0.3927829266 0.9070000000 0.0008499622

131 1 windows install 0.3926658630 0.9100000000 0.0006680489

132 1 windows install 0.4005849361 0.9140000000 0.0000658035

133 1 windows install 0.3947939873 0.7060000000 0.0012660027

134 1 windows install 0.3988029957 0.9180000000 0.0007231236

135 1 windows install 0.3888220787 0.7400000000 0.0012609959

136 1 windows install 0.3907008171 0.9340000000 0.0005657673

Page 69: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXV

137 1 windows install 0.4007120132 0.7560000000 0.0000479221

138 1 windows install 0.3851540089 0.9130000000 0.0025401115

139 1 windows install 0.1870942116 0.4820000000 0.0015361309

140 1 windows install 0.3928709030 0.9210000000 0.0011370182

141 1 windows install 0.4008240700 0.6890000000 0.0000228882

142 1 windows install 0.3947658539 0.9070000000 0.0009758472

143 1 windows install 0.1907060146 0.7350000000 0.0014717579

144 1 windows install 0.1962409019 0.7250000000 0.1080930233

145 1 windows install 0.3926041126 0.9150000000 0.0006101131

146 1 windows install 0.4009490013 0.9530000000 0.0004141331

147 1 windows install 0.3934738636 0.9250000000 0.0013160706

148 1 windows install 0.3922188282 0.9070000000 0.0007688999

149 1 windows install 0.3912689686 0.9110000000 0.0012679100

150 1 windows install 0.3911750317 0.7010000000 0.0012190342

151 1 windows install 0.1898350716 0.7100000000 0.0011999607

152 1 windows install 0.3925390244 0.9080000000 0.0006222725

153 1 windows install 0.3913710117 0.8980000000 0.0011529922

154 1 windows install 0.3918991089 0.9110000000 0.0008571148

155 1 windows install 0.3923871517 0.8950000000 0.0006859303

156 1 windows install 0.1865470409 0.7030000000 0.0004098415

157 1 windows install 0.3939578533 0.7390000000 0.0009410381

158 1 windows install 0.3928940296 0.9110000000 0.0012409687

159 1 windows install 0.3965511322 0.9070000000 0.0006170273

160 1 windows install 0.3845608234 0.7350000000 0.0025122166

161 1 windows install 0.1910409927 0.7100000000 0.0018029213

162 1 windows install 0.3885180950 0.9130000000 0.0007190704

163 1 windows install 0.3924109936 0.9540000000 0.0006678104

164 1 windows install 0.3903381824 0.9020000000 0.0004241467

165 1 windows install 0.3910501003 0.9200000000 0.0011630058

166 1 windows install 0.3907651901 0.9130000000 0.0006489754

167 1 windows install 0.3911380768 0.9540000000 0.0012221336

168 1 windows install 0.3971469402 0.7070000000 0.0009269714

169 1 windows install 0.3948428631 0.9090000000 0.0013980865

170 1 windows install 0.3976209164 0.9150000000 0.0007688999

171 1 windows install 0.1888079643 0.7230000000 0.0017862320

172 1 windows install 0.3902449608 0.9140000000 0.0004129410

173 1 windows install 0.4007790089 0.9500000000 0.0000240803

174 1 windows install 0.3909838200 0.9530000000 0.0011148453

175 1 windows install 0.3925080299 0.9250000000 0.0006120205

176 1 windows install 0.3929550648 0.9110000000 0.0014550686

177 1 windows install 0.3887598515 0.9120000000 0.0011589527

178 1 windows install 0.3911759853 0.7120000000 0.0012111664

179 1 windows install 0.3935608864 0.9280000000 0.0012729168

180 1 windows install 0.3905429840 0.9170000000 0.0004050732

181 1 windows install 0.3893380165 0.9180000000 0.0011482239

182 1 windows install 0.1989188194 0.7400000000 0.1110692024

183 1 windows install 0.3880000114 0.9170000000 0.0008778572

184 1 windows install 0.1928920746 0.7170000000 0.0037071705

Page 70: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXVI

185 1 windows install 0.3928840160 0.9200000000 0.0012121201

186 1 windows install 0.3922209740 0.9150000000 0.0007600784

187 1 windows install 0.3965590000 0.9390000000 0.0006101131

188 1 windows install 0.3979058266 0.9160000000 0.0006299019

189 1 windows install 0.3900239468 0.9310000000 0.0005290508

190 1 windows install 0.3947639465 0.9490000000 0.0009710789

191 1 windows install 0.1921970844 0.7060000000 0.0022401810

192 1 windows install 0.3887479305 0.9280000000 0.0011119843

193 1 windows install 0.3936929703 0.9240000000 0.0011761189

194 1 windows install 0.1924140453 0.7390000000 0.0024771690

195 1 windows install 0.4046490192 0.9270000000 0.0004389286

196 1 windows install 0.1900382042 0.7230000000 0.0011060238

197 1 windows install 0.3865249157 0.8900000000 0.0018959045

198 1 windows install 0.3821120262 0.9050000000 0.0021400452

199 1 windows install 0.1926200390 0.5120000000 0.0035068989

200 1 windows install 0.3835420609 0.9250000000 0.0031528473

201 1 java get time 0.3922219276 0.9350000000 0.0006630421

202 1 java get time 0.3989689350 0.9360000000 0.0007038116

203 1 java get time 0.1925058365 0.4810000000 0.0026500225

204 1 java get time 0.3920569420 0.9230000000 0.0007789135

205 1 java get time 0.3908479214 0.9350000000 0.0008108616

206 1 java get time 0.3807601929 0.9140000000 0.0027918816

207 1 java get time 0.3876600266 0.9240000000 0.0012362003

208 1 java get time 0.3889229298 0.9160000000 0.0012168884

209 1 java get time 0.3866679668 0.9010000000 0.0019199848

210 1 java get time 0.1907000542 0.7390000000 0.0014150143

211 1 java get time 0.3960819244 0.9340000000 0.0007038116

212 1 java get time 0.3923451900 0.9330000000 0.0006809235

213 1 java get time 0.3905158043 0.9170000000 0.0003838539

214 1 java get time 0.3928120136 0.9180000000 0.0009310246

215 1 java get time 0.1883981228 0.7020000000 0.0016083717

216 1 java get time 0.3908569813 0.7000000000 0.0008230209

217 1 java get time 0.3847777843 0.9330000000 0.0022618771

218 1 java get time 0.3901600838 0.7030000000 0.0004870892

219 1 java get time 0.2046930790 0.7280000000 0.1065561771

220 1 java get time 0.3926899433 0.9120000000 0.0006599426

221 1 java get time 0.3912839890 0.9120000000 0.0011940002

222 1 java get time 0.3947410583 0.9160000000 0.0009210110

223 1 java get time 0.1869261265 0.5250000000 0.0015778542

224 1 java get time 0.1887838840 0.4930000000 0.0017302036

225 1 java get time 0.3923530579 0.9240000000 0.0006599426

226 1 java get time 0.3925189972 0.9230000000 0.0006239414

227 1 java get time 0.3909091949 0.9080000000 0.0008699894

228 1 java get time 0.3998589516 0.7360000000 0.0004310608

229 1 java get time 0.3927180767 0.9230000000 0.0006899834

230 1 java get time 0.3825840950 0.7140000000 0.0030930042

231 1 java get time 0.3988540173 0.9050000000 0.0006911755

232 1 java get time 0.3931808472 0.9200000000 0.0013892651

Page 71: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXVII

233 1 java get time 0.3920130730 0.9150000000 0.0008060932

234 1 java get time 0.3891220093 0.9320000000 0.0011968613

235 1 java get time 0.1846392155 0.7170000000 0.0018720627

236 1 java get time 0.1897361279 0.4900000000 0.0011079311

237 1 java get time 0.3891050816 0.7260000000 0.0011970997

238 1 java get time 0.3885359764 0.7250000000 0.0007550716

239 1 java get time 0.3873448372 0.9100000000 0.0013060570

240 1 java get time 0.1917769909 0.7010000000 0.0018551350

241 1 java get time 0.3886051178 0.9150000000 0.0008089542

242 1 java get time 0.1842601299 0.5320000000 0.0019371510

243 1 java get time 0.3866610527 0.8950000000 0.0019071102

244 1 java get time 0.3937859535 0.9110000000 0.0010151863

245 1 java get time 0.1930861473 0.7350000000 0.0040388107

246 1 java get time 0.3930330276 0.7270000000 0.0014853477

247 1 java get time 0.1859869957 0.5270000000 0.0007929802

248 1 java get time 0.1920471191 0.5240000000 0.0023517609

249 1 java get time 0.3887920380 0.6960000000 0.0011990070

250 1 java get time 0.3831448555 0.9160000000 0.0034000874

251 1 java get time 0.3928010464 0.9100000000 0.0008540154

252 1 java get time 0.1986899376 0.7040000000 0.1092708111

253 1 java get time 0.3914248943 0.9080000000 0.0011470318

254 1 java get time 0.1886820793 0.7110000000 0.0016930103

255 1 java get time 0.3907630444 0.9030000000 0.0005850792

256 1 java get time 0.3867459297 0.9120000000 0.0017831326

257 1 java get time 0.3872859478 0.9010000000 0.0013980865

258 1 java get time 0.3907029629 0.9140000000 0.0005719662

259 1 java get time 0.3886840343 0.9090000000 0.0009949207

260 1 java get time 0.3910870552 0.9160000000 0.0011928082

261 1 java get time 0.3905858994 0.9190000000 0.0004401207

262 1 java get time 0.3866748810 0.7160000000 0.0019190311

263 1 java get time 0.3887040615 0.9010000000 0.0010099411

264 1 java get time 0.1873240471 0.7020000000 0.0014541149

265 1 java get time 0.3886809349 0.9260000000 0.0009200573

266 1 java get time 0.3810229301 0.9110000000 0.0025501251

267 1 java get time 0.3867330551 0.9070000000 0.0018980503

268 1 java get time 0.3943850994 0.9170000000 0.0006697178

269 1 java get time 0.3948149681 0.7000000000 0.0014081001

270 1 java get time 0.3925349712 0.9270000000 0.0006220341

271 1 java get time 0.3888800144 0.8870000000 0.0012540817

272 1 java get time 0.3908951283 0.9080000000 0.0008027554

273 1 java get time 0.3904540539 0.9240000000 0.0003612041

274 1 java get time 0.3906750679 0.9220000000 0.0005071163

275 1 java get time 0.3882009983 0.7140000000 0.0008161068

276 1 java get time 0.1868391037 0.7050000000 0.0015759468

277 1 java get time 0.3911499977 0.7010000000 0.0012180805

278 1 java get time 0.3910501003 0.9160000000 0.0011999607

279 1 java get time 0.3929471970 0.9150000000 0.0014479160

280 1 java get time 0.3851909637 0.9210000000 0.0023510456

Page 72: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXVIII

281 1 java get time 0.3930559158 0.6970000000 0.0014719963

282 1 java get time 0.1967840195 0.7120000000 0.1092851162

283 1 java get time 0.3925440311 0.7160000000 0.0006210804

284 1 java get time 0.3896141052 0.7060000000 0.0008800030

285 1 java get time 0.3892850876 0.9110000000 0.0011889935

286 1 java get time 0.3904531002 0.9620000000 0.0003798008

287 1 java get time 0.3903548717 0.9140000000 0.0003929138

288 1 java get time 0.4106230736 0.9170000000 0.0027458668

289 1 java get time 0.4048030376 0.9350000000 0.0004758835

290 1 java get time 0.3929221630 0.9130000000 0.0013101101

291 1 java get time 0.3954970837 0.9530000000 0.0011439323

292 1 java get time 0.3996870518 0.9100000000 0.0004668236

293 1 java get time 0.1868119240 0.7150000000 0.0013239384

294 1 java get time 0.3904860020 0.9230000000 0.0003681183

295 1 java get time 0.3831520081 0.8990000000 0.0032410622

296 1 java get time 0.3901681900 0.7160000000 0.0005190372

297 1 java get time 0.1934318542 0.7240000000 0.0053508282

298 1 java get time 0.3870229721 0.6960000000 0.0016210079

299 1 java get time 0.2008318901 0.6930000000 0.1099503040

300 1 java get time 0.3928120136 0.9160000000 0.0009291172

301 1 java get time 0.1890561581 0.7180000000 0.0015828609

302 1 java get time 0.3936369419 0.9040000000 0.0012118816

303 1 java get time 0.4026539326 0.9360000000 0.0003137589

304 1 java get time 0.1927399635 0.5480000000 0.0036818981

305 1 java get time 0.1924829483 0.6730000000 0.0026631355

306 1 java get time 0.1885387897 0.7080000000 0.0017352104

307 1 java get time 0.3957128525 0.7260000000 0.0009889603

308 1 java get time 0.3841998577 0.8570000000 0.0023970604

309 1 java get time 0.3884048462 0.9530000000 0.0006451607

310 1 java get time 0.3928518295 0.9220000000 0.0010342598

311 1 java get time 0.3919999599 0.9030000000 0.0008251667

312 1 java get time 0.3932590485 0.9280000000 0.0013959408

313 1 java get time 0.3928189278 0.9050000000 0.0009677410

314 1 java get time 0.3934700489 0.9180000000 0.0013139248

315 1 java get time 0.3937849998 0.9250000000 0.0010581017

316 1 java get time 0.1908340454 0.7240000000 0.0017831326

317 1 java get time 0.3986880779 0.9230000000 0.0003979206

318 1 java get time 0.4123330116 0.9360000000 0.0032937527

319 1 java get time 0.3948078156 0.9290000000 0.0014150143

320 1 java get time 0.3905467987 0.9220000000 0.0004029274

321 1 java get time 0.3837530613 0.7210000000 0.0026819706

322 1 java get time 0.3889408112 0.9080000000 0.0012710094

323 1 java get time 0.3869760036 0.9160000000 0.0016219616

324 1 java get time 0.3866939545 0.9420000000 0.0018939972

325 1 java get time 0.3847739697 0.9430000000 0.0022108555

326 1 java get time 0.3915240765 0.9190000000 0.0010831356

327 1 java get time 0.3945138454 0.9070000000 0.0006618500

328 1 java get time 0.3936779499 0.9160000000 0.0012128353

Page 73: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XXXIX

329 1 java get time 0.3929309845 0.9370000000 0.0013408661

330 1 java get time 0.1866011620 0.7250000000 0.0005240440

331 1 java get time 0.3892810345 0.9250000000 0.0011899471

332 1 java get time 0.3950340748 0.9000000000 0.0014791489

333 1 java get time 0.3956639767 0.8940000000 0.0010700226

334 1 java get time 0.3948030472 0.9210000000 0.0014400482

335 1 java get time 0.3972799778 0.9190000000 0.0008571148

336 1 java get time 0.3918190002 0.9060000000 0.0009360313

337 1 java get time 0.3909490108 0.9140000000 0.0010681152

338 1 java get time 0.3949239254 0.9230000000 0.0015099049

339 1 java get time 0.4032628536 0.9110000000 0.0004928112

340 1 java get time 0.3886649609 0.9240000000 0.0009090900

341 1 java get time 0.1886150837 0.7040000000 0.0016770363

342 1 java get time 0.1925740242 0.7040000000 0.0027730465

343 1 java get time 0.3925030231 0.9100000000 0.0006191730

344 1 java get time 0.3909180164 0.9000000000 0.0008671284

345 1 java get time 0.3922379017 0.9500000000 0.0006649494

346 1 java get time 0.3949930668 0.7250000000 0.0015199184

347 1 java get time 0.1855988503 0.6990000000 0.0010850430

348 1 java get time 0.3890669346 0.7220000000 0.0012159348

349 1 java get time 0.3911969662 0.7020000000 0.0012409687

350 1 java get time 0.1909329891 0.7310000000 0.0019810200

351 1 java get time 0.3905510902 0.9080000000 0.0004000664

352 1 java get time 0.3940279484 0.9220000000 0.0009169579

353 1 java get time 0.3950150013 0.9250000000 0.0014989376

354 1 java get time 0.3928129673 0.8950000000 0.0009329319

355 1 java get time 0.4025831223 0.9230000000 0.0005950928

356 1 java get time 0.3942251205 0.9240000000 0.0007719994

357 1 java get time 0.1861150265 0.6890000000 0.0004420280

358 1 java get time 0.3906848431 0.9070000000 0.0005102158

359 1 java get time 0.3886251450 0.9050000000 0.0008049011

360 1 java get time 0.3887851238 0.9300000000 0.0011529922

361 1 java get time 0.3907082081 0.9140000000 0.0005481243

362 1 java get time 0.3948860168 0.9240000000 0.0015428066

363 1 java get time 0.3935229778 0.6980000000 0.0012848377

364 1 java get time 0.3887801170 0.9490000000 0.0011658669

365 1 java get time 0.3925108910 0.9270000000 0.0005970001

366 1 java get time 0.3929100037 0.9250000000 0.0012989044

367 1 java get time 0.3929078579 0.9090000000 0.0012929440

368 1 java get time 0.3866770267 0.8910000000 0.0019161701

369 1 java get time 0.1887331009 0.5120000000 0.0017139912

370 1 java get time 0.1870748997 0.5120000000 0.0015177727

371 1 java get time 0.3915009499 0.9480000000 0.0011410713

372 1 java get time 0.3907849789 0.8970000000 0.0006999969

373 1 java get time 0.3825500011 0.9120000000 0.0028412342

374 1 java get time 0.3927299976 0.9060000000 0.0007369518

375 1 java get time 0.3909289837 0.9230000000 0.0009920597

376 1 java get time 0.3829810619 0.9110000000 0.0032699108

Page 74: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XL

377 1 java get time 0.3956248760 0.9190000000 0.0010960102

378 1 java get time 0.1909089088 0.5100000000 0.0018880367

379 1 java get time 0.3866829872 0.9060000000 0.0019168854

380 1 java get time 0.4041349888 0.9280000000 0.0009050369

381 1 java get time 0.1906578541 0.7020000000 0.0014338493

382 1 java get time 0.3914909363 0.9140000000 0.0011451244

383 1 java get time 0.3885929585 0.7400000000 0.0007767677

384 1 java get time 0.1884818077 0.7080000000 0.0016880035

385 1 java get time 0.4023039341 0.9140000000 0.0009651184

386 1 java get time 0.3988969326 0.7450000000 0.0007121563

387 1 java get time 0.3927779198 0.9230000000 0.0008151531

388 1 java get time 0.3909850121 0.9090000000 0.0011470318

389 1 java get time 0.3929789066 0.9230000000 0.0014450550

390 1 java get time 0.3919081688 0.9120000000 0.0007979870

391 1 java get time 0.3868749142 0.9010000000 0.0016849041

392 1 java get time 0.3987529278 0.9030000000 0.0006570816

393 1 java get time 0.3887579441 0.9110000000 0.0011708736

394 1 java get time 0.3888189793 0.7070000000 0.0012531281

395 1 java get time 0.1868600845 0.5240000000 0.0015318394

396 1 java get time 0.3929119110 0.6950000000 0.0012910366

397 1 java get time 0.3829269409 0.9260000000 0.0031630993

398 1 java get time 0.1906521320 0.6940000000 0.0012562275

399 1 java get time 0.3845560551 0.9080000000 0.0025310516

400 1 java get time 0.1844179630 0.7520000000 0.0018858910

401 1 JavaScript Cookies 0.3178958893 0.8270000000 0.0045950413

402 1 JavaScript Cookies 0.1141297817 0.6250000000 0.0342390537

403 1 JavaScript Cookies 0.3103649616 0.8390000000 0.0028631687

404 1 JavaScript Cookies 0.3071279526 0.8150000000 0.0061190128

405 1 JavaScript Cookies 0.3187518120 0.6100000000 0.0050258636

406 1 JavaScript Cookies 0.3221638203 0.8580000000 0.0098109245

407 1 JavaScript Cookies 0.1119749546 0.6140000000 0.0016429424

408 1 JavaScript Cookies 0.1140758991 0.6420000000 0.0341100693

409 1 JavaScript Cookies 0.3158800602 0.6230000000 0.0025420189

410 1 JavaScript Cookies 0.3154919147 0.6340000000 0.0023539066

411 1 JavaScript Cookies 0.1118309498 0.6400000000 0.0003170967

412 1 JavaScript Cookies 0.3143029213 0.8320000000 0.0024731159

413 1 JavaScript Cookies 0.1145169735 0.6390000000 0.0340709686

414 1 JavaScript Cookies 0.3138370514 0.8140000000 0.0019237995

415 1 JavaScript Cookies 0.3190150261 0.8520000000 0.0043017864

416 1 JavaScript Cookies 0.3076519966 0.8180000000 0.0057539940

417 1 JavaScript Cookies 0.3221487999 0.8440000000 0.0096843243

418 1 JavaScript Cookies 0.3137447834 0.6450000000 0.0019149780

419 1 JavaScript Cookies 0.3144760132 0.8380000000 0.0024392605

420 1 JavaScript Cookies 0.1132199764 0.6410000000 0.0023078918

421 1 JavaScript Cookies 0.3141469955 0.8220000000 0.0021839142

422 1 JavaScript Cookies 0.3161680698 0.6400000000 0.0035228729

423 1 JavaScript Cookies 0.3157811165 0.6430000000 0.0023012161

424 1 JavaScript Cookies 0.3121011257 0.8460000000 0.0023031235

Page 75: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLI

425 1 JavaScript Cookies 0.1197881699 0.6220000000 0.0298678875

426 1 JavaScript Cookies 0.3163568974 0.8300000000 0.0034301281

427 1 JavaScript Cookies 0.3141739368 0.6390000000 0.0023341179

428 1 JavaScript Cookies 0.3138828278 0.8440000000 0.0018749237

429 1 JavaScript Cookies 0.1119780540 0.6340000000 0.0018320084

430 1 JavaScript Cookies 0.3125450611 0.8280000000 0.0023608208

431 1 JavaScript Cookies 0.3054850101 0.7970000000 0.0061740875

432 1 JavaScript Cookies 0.3257901669 0.6520000000 0.0125620365

433 1 JavaScript Cookies 0.3229711056 0.8380000000 0.0120990276

434 1 JavaScript Cookies 0.3166308403 0.8470000000 0.0032401085

435 1 JavaScript Cookies 0.3147230148 0.8510000000 0.0024631023

436 1 JavaScript Cookies 0.3141388893 0.8410000000 0.0021820068

437 1 JavaScript Cookies 0.1126527786 0.6450000000 0.0014791489

438 1 JavaScript Cookies 0.3160710335 0.8280000000 0.0028769970

439 1 JavaScript Cookies 0.3164789677 0.8410000000 0.0033512115

440 1 JavaScript Cookies 0.3041210175 0.8380000000 0.0078921318

441 1 JavaScript Cookies 0.3087539673 0.8060000000 0.0038428307

442 1 JavaScript Cookies 0.3041589260 0.8300000000 0.0065770149

443 1 JavaScript Cookies 0.3155670166 0.8180000000 0.0023999214

444 1 JavaScript Cookies 0.3117740154 0.8120000000 0.0022428036

445 1 JavaScript Cookies 0.3060050011 0.8200000000 0.0060911179

446 1 JavaScript Cookies 0.3159351349 0.8380000000 0.0027170181

447 1 JavaScript Cookies 0.3118669987 0.8090000000 0.0022599697

448 1 JavaScript Cookies 0.3178780079 0.8330000000 0.0043687820

449 1 JavaScript Cookies 0.3161010742 0.8150000000 0.0034821033

450 1 JavaScript Cookies 0.3137011528 0.6460000000 0.0020053387

451 1 JavaScript Cookies 0.3076510429 0.8400000000 0.0061230659

452 1 JavaScript Cookies 0.3178880215 0.8100000000 0.0046513081

453 1 JavaScript Cookies 0.3146369457 0.8440000000 0.0028939247

454 1 JavaScript Cookies 0.3141219616 0.8390000000 0.0021688938

455 1 JavaScript Cookies 0.3201050758 0.8540000000 0.0043518543

456 1 JavaScript Cookies 0.3140490055 0.8270000000 0.0020549297

457 1 JavaScript Cookies 0.3299899101 0.8500000000 0.0122437477

458 1 JavaScript Cookies 0.1149480343 0.6290000000 0.0334579945

459 1 JavaScript Cookies 0.3222179413 0.6560000000 0.0105540752

460 1 JavaScript Cookies 0.3155150414 0.8280000000 0.0023431778

461 1 JavaScript Cookies 0.3158907890 0.8380000000 0.0025470257

462 1 JavaScript Cookies 0.1149249077 0.6380000000 0.0341038704

463 1 JavaScript Cookies 0.3140971661 0.6320000000 0.0020630360

464 1 JavaScript Cookies 0.1116828918 0.6100000000 0.0000476837

465 1 JavaScript Cookies 0.3140790462 0.6310000000 0.0020618439

466 1 JavaScript Cookies 0.1101989746 0.6200000000 0.0006668568

467 1 JavaScript Cookies 0.3141438961 0.8370000000 0.0021932125

468 1 JavaScript Cookies 0.1158978939 0.4140000000 0.0332498550

469 1 JavaScript Cookies 0.3060729504 0.6390000000 0.0062298775

470 1 JavaScript Cookies 0.3130929470 0.8240000000 0.0019938946

471 1 JavaScript Cookies 0.1123571396 0.6250000000 0.0017092228

472 1 JavaScript Cookies 0.3066108227 0.8250000000 0.0056791306

Page 76: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLII

473 1 JavaScript Cookies 0.1120319366 0.6320000000 0.0017838478

474 1 JavaScript Cookies 0.1129069328 0.6360000000 0.0023729801

475 1 JavaScript Cookies 0.3243231773 0.8420000000 0.0161349773

476 1 JavaScript Cookies 0.3254730701 0.6300000000 0.0167520046

477 1 JavaScript Cookies 0.1139051914 0.6230000000 0.0061888695

478 1 JavaScript Cookies 0.3219451904 0.8480000000 0.0055351257

479 1 JavaScript Cookies 0.1119060516 0.6370000000 0.0014503002

480 1 JavaScript Cookies 0.3167390823 0.8400000000 0.0034651756

481 1 JavaScript Cookies 0.3178260326 0.8390000000 0.0042998791

482 1 JavaScript Cookies 0.1138241291 0.6190000000 0.0024580956

483 1 JavaScript Cookies 0.3159060478 0.8450000000 0.0025417805

484 1 JavaScript Cookies 0.3161289692 0.8430000000 0.0035219193

485 1 JavaScript Cookies 0.3122029305 0.8300000000 0.0023350716

486 1 JavaScript Cookies 0.3143861294 0.8240000000 0.0024929047

487 1 JavaScript Cookies 0.1139149666 0.6350000000 0.0339589119

488 1 JavaScript Cookies 0.1262061596 0.4330000000 0.0023748875

489 1 JavaScript Cookies 0.3157148361 0.8240000000 0.0023438931

490 1 JavaScript Cookies 0.3155920506 0.6300000000 0.0023860931

491 1 JavaScript Cookies 0.3142039776 0.8070000000 0.0024421215

492 1 JavaScript Cookies 0.3140058517 0.8420000000 0.0020458698

493 1 JavaScript Cookies 0.1123929024 0.6220000000 0.0015790462

494 1 JavaScript Cookies 0.3139529228 0.6340000000 0.0019762516

495 1 JavaScript Cookies 0.1115629673 0.6060000000 0.0001187325

496 1 JavaScript Cookies 0.3141279221 0.8260000000 0.0021760464

497 1 JavaScript Cookies 0.3142249584 0.8060000000 0.0024821758

498 1 JavaScript Cookies 0.3120470047 0.8270000000 0.0022900105

499 1 JavaScript Cookies 0.3136458397 0.8250000000 0.0019321442

500 1 JavaScript Cookies 0.3184459209 0.8250000000 0.0047750473

501 1 JavaScript Cookies 0.3143889904 0.8430000000 0.0024960041

502 1 JavaScript Cookies 0.3172559738 0.6250000000 0.0033190250

503 1 JavaScript Cookies 0.3167169094 0.8390000000 0.0032198429

504 1 JavaScript Cookies 0.3318719864 0.8620000000 0.0068240166

505 1 JavaScript Cookies 0.3139009476 0.8060000000 0.0019421577

506 1 JavaScript Cookies 0.3232810497 0.8520000000 0.0123941898

507 1 JavaScript Cookies 0.3140580654 0.8430000000 0.0020470619

508 1 JavaScript Cookies 0.3129341602 0.8440000000 0.0021808147

509 1 JavaScript Cookies 0.3140361309 0.8270000000 0.0020389557

510 1 JavaScript Cookies 0.3168008327 0.8350000000 0.0033841133

511 1 JavaScript Cookies 0.3067219257 0.8380000000 0.0055570602

512 1 JavaScript Cookies 0.3173239231 0.8370000000 0.0038378239

513 1 JavaScript Cookies 0.3127331734 0.8180000000 0.0022187233

514 1 JavaScript Cookies 0.3362820148 0.8680000000 0.0005979538

515 1 JavaScript Cookies 0.3122241497 0.8170000000 0.0024027824

516 1 JavaScript Cookies 0.3218669891 0.8490000000 0.0046889782

517 1 JavaScript Cookies 0.3100991249 0.8430000000 0.0035429001

518 1 JavaScript Cookies 0.3199820518 0.8290000000 0.0041940212

519 1 JavaScript Cookies 0.1200439930 0.6330000000 0.0298089981

520 1 JavaScript Cookies 0.3070309162 0.6050000000 0.0064680576

Page 77: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLIII

521 1 JavaScript Cookies 0.3122148514 0.8320000000 0.0023450851

522 1 JavaScript Cookies 0.3219549656 0.8480000000 0.0061199665

523 1 JavaScript Cookies 0.1135680676 0.6420000000 0.0022437572

524 1 JavaScript Cookies 0.3141040802 0.8320000000 0.0020668507

525 1 JavaScript Cookies 0.1125800610 0.4280000000 0.0017082691

526 1 JavaScript Cookies 0.3145818710 0.8220000000 0.0027790070

527 1 JavaScript Cookies 0.3120999336 0.8220000000 0.0023918152

528 1 JavaScript Cookies 0.3130040169 0.8210000000 0.0021462440

529 1 JavaScript Cookies 0.3140261173 0.8330000000 0.0020408630

530 1 JavaScript Cookies 0.3152971268 0.8400000000 0.0023591518

531 1 JavaScript Cookies 0.3155558109 0.8290000000 0.0023801327

532 1 JavaScript Cookies 0.3141629696 0.8290000000 0.0022659302

533 1 JavaScript Cookies 0.1116051674 0.6010000000 0.0001010895

534 1 JavaScript Cookies 0.3157439232 0.8410000000 0.0023429394

535 1 JavaScript Cookies 0.3103389740 0.6480000000 0.0035250187

536 1 JavaScript Cookies 0.3119750023 0.8360000000 0.0023038387

537 1 JavaScript Cookies 0.3304750919 0.6390000000 0.0108749866

538 1 JavaScript Cookies 0.3158710003 0.6290000000 0.0025479794

539 1 JavaScript Cookies 0.3122169971 0.6450000000 0.0024859905

540 1 JavaScript Cookies 0.3138411045 0.8330000000 0.0018861294

541 1 JavaScript Cookies 0.3038151264 0.8080000000 0.0079281330

542 1 JavaScript Cookies 0.1138908863 0.6140000000 0.0049169064

543 1 JavaScript Cookies 0.3100290298 0.8130000000 0.0037739277

544 1 JavaScript Cookies 0.1118519306 0.6570000000 0.0007247925

545 1 JavaScript Cookies 0.1119558811 0.6220000000 0.0014441013

546 1 JavaScript Cookies 0.3117098808 0.8550000000 0.0022850037

547 1 JavaScript Cookies 0.3183200359 0.6310000000 0.0045189857

548 1 JavaScript Cookies 0.1095571518 0.6190000000 0.0001008511

549 1 JavaScript Cookies 0.3122220039 0.6270000000 0.0025360584

550 1 JavaScript Cookies 0.3165729046 0.8040000000 0.0033001900

551 1 JavaScript Cookies 0.3039658070 0.7960000000 0.0079138279

552 1 JavaScript Cookies 0.3242161274 0.6750000000 0.0164120197

553 1 JavaScript Cookies 0.3139109612 0.8310000000 0.0020241737

554 1 JavaScript Cookies 0.3143589497 0.8390000000 0.0024497509

555 1 JavaScript Cookies 0.3061120510 0.8280000000 0.0057492256

556 1 JavaScript Cookies 0.3059289455 0.8190000000 0.0061480999

557 1 JavaScript Cookies 0.3158478737 0.8420000000 0.0023961067

558 1 JavaScript Cookies 0.3111238480 0.8270000000 0.0025029182

559 1 JavaScript Cookies 0.3141150475 0.8200000000 0.0021271706

560 1 JavaScript Cookies 0.3219969273 0.8530000000 0.0076062679

561 1 JavaScript Cookies 0.3123731613 0.6410000000 0.0023782253

562 1 JavaScript Cookies 0.3156278133 0.8420000000 0.0023200512

563 1 JavaScript Cookies 0.3141610622 0.8300000000 0.0022430420

564 1 JavaScript Cookies 0.3209829330 0.8390000000 0.0041642189

565 1 JavaScript Cookies 0.3159399033 0.8400000000 0.0026209354

566 1 JavaScript Cookies 0.3118700981 0.8290000000 0.0023300648

567 1 JavaScript Cookies 0.3161931038 0.6380000000 0.0034720898

568 1 JavaScript Cookies 0.1117541790 0.6280000000 0.0001580715

Page 78: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLIV

569 1 JavaScript Cookies 0.3140840530 0.8280000000 0.0020649433

570 1 JavaScript Cookies 0.3141169548 0.8380000000 0.0021450520

571 1 JavaScript Cookies 0.1160540581 0.4310000000 0.0301361084

572 1 JavaScript Cookies 0.3120679855 0.8240000000 0.0023341179

573 1 JavaScript Cookies 0.1201269627 0.6400000000 0.0241017342

574 1 JavaScript Cookies 0.3141250610 0.8280000000 0.0021789074

575 1 JavaScript Cookies 0.3162519932 0.8170000000 0.0033712387

576 1 JavaScript Cookies 0.3218069077 0.8560000000 0.0042557716

577 1 JavaScript Cookies 0.3141329288 0.8470000000 0.0021800995

578 1 JavaScript Cookies 0.1141059399 0.6170000000 0.0344502926

579 1 JavaScript Cookies 0.3140568733 0.8400000000 0.0020589828

580 1 JavaScript Cookies 0.3238267899 0.8460000000 0.0147047043

581 1 JavaScript Cookies 0.3155040741 0.8470000000 0.0023579597

582 1 JavaScript Cookies 0.1157741547 0.6100000000 0.0333859921

583 1 JavaScript Cookies 0.3250460625 0.8300000000 0.0166108608

584 1 JavaScript Cookies 0.3201200962 0.8570000000 0.0035500526

585 1 JavaScript Cookies 0.3134140968 0.8300000000 0.0018780231

586 1 JavaScript Cookies 0.3119018078 0.8230000000 0.0023519993

587 1 JavaScript Cookies 0.1121370792 0.6230000000 0.0017259121

588 1 JavaScript Cookies 0.3115069866 0.8400000000 0.0023353100

589 1 JavaScript Cookies 0.3136310577 0.8370000000 0.0019083023

590 1 JavaScript Cookies 0.1195549965 0.6330000000 0.0300869942

591 1 JavaScript Cookies 0.3066818714 0.8280000000 0.0058391094

592 1 JavaScript Cookies 0.3136689663 0.8350000000 0.0019629002

593 1 JavaScript Cookies 0.1125459671 0.6250000000 0.0016269684

594 1 JavaScript Cookies 0.3124248981 0.8360000000 0.0024459362

595 1 JavaScript Cookies 0.3145458698 0.8340000000 0.0025310516

596 1 JavaScript Cookies 0.3160631657 0.6300000000 0.0026800632

597 1 JavaScript Cookies 0.3218071461 0.8680000000 0.0043091774

598 1 JavaScript Cookies 0.3141958714 0.8280000000 0.0023441315

599 1 JavaScript Cookies 0.3080580235 0.6330000000 0.0050950050

600 1 JavaScript Cookies 0.3151779175 0.6340000000 0.0024588108

601 2 windows install 0.1905748844 0.7570000000 0.0012652874

602 2 windows install 0.3886430264 0.9320000000 0.0008711815

603 2 windows install 0.1907401085 0.7440000000 0.0014979839

604 2 windows install 0.3993849754 0.9200000000 0.0006058216

605 2 windows install 0.1927380562 0.7190000000 0.0037720203

606 2 windows install 0.3905301094 0.9400000000 0.0003938675

607 2 windows install 0.3875899315 0.9030000000 0.0012469292

608 2 windows install 0.3945770264 0.9030000000 0.0006918907

609 2 windows install 0.3971841335 0.9240000000 0.0009322166

610 2 windows install 0.1926510334 0.7070000000 0.0036721230

611 2 windows install 0.3860778809 0.9040000000 0.0020000935

612 2 windows install 0.3912098408 0.9180000000 0.0012390614

613 2 windows install 0.3823390007 0.9310000000 0.0024690628

614 2 windows install 0.3901169300 0.9120000000 0.0005090237

615 2 windows install 0.1948781013 0.7520000000 0.0073449612

616 2 windows install 0.4004189968 0.9370000000 0.0000209808

Page 79: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLV

617 2 windows install 0.3926539421 0.9010000000 0.0006411076

618 2 windows install 0.3911659718 0.9130000000 0.0012190342

619 2 windows install 0.3986890316 0.9380000000 0.0003900528

620 2 windows install 0.3936128616 0.9180000000 0.0012450218

621 2 windows install 0.1891520023 0.5080000000 0.0012340546

622 2 windows install 0.3901820183 0.9280000000 0.0004651546

623 2 windows install 0.1966850758 0.7070000000 0.1092967987

624 2 windows install 0.3927769661 0.9420000000 0.0007729530

625 2 windows install 0.3921871185 0.9130000000 0.0007100105

626 2 windows install 0.1963689327 0.7490000000 0.1092648506

627 2 windows install 0.3928527832 0.9260000000 0.0010821819

628 2 windows install 0.3946788311 0.7160000000 0.0007688999

629 2 windows install 0.1895549297 0.7330000000 0.0011870861

630 2 windows install 0.3915569782 0.7160000000 0.0010468960

631 2 windows install 0.3859500885 0.7050000000 0.0020189285

632 2 windows install 0.3923971653 0.9380000000 0.0006821156

633 2 windows install 0.3894269466 0.9190000000 0.0009710789

634 2 windows install 0.1902689934 0.7130000000 0.0008738041

635 2 windows install 0.3950531483 0.9450000000 0.0015158653

636 2 windows install 0.3925600052 0.9510000000 0.0006239414

637 2 windows install 0.3914418221 0.7070000000 0.0011398792

638 2 windows install 0.3893668652 0.9120000000 0.0011050701

639 2 windows install 0.3903369904 0.8880000000 0.0004119873

640 2 windows install 0.3925979137 0.9130000000 0.0006048679

641 2 windows install 0.4002010822 0.9150000000 0.0001730919

642 2 windows install 0.3920960426 0.9170000000 0.0007529259

643 2 windows install 0.3905260563 0.7240000000 0.0003800392

644 2 windows install 0.3866059780 0.6730000000 0.0019049644

645 2 windows install 0.3907089233 0.9240000000 0.0005569458

646 2 windows install 0.3903329372 0.9040000000 0.0004208088

647 2 windows install 0.1925809383 0.6910000000 0.0030696392

648 2 windows install 0.3947179317 0.9270000000 0.0008099079

649 2 windows install 0.3986108303 0.9120000000 0.0003938675

650 2 windows install 0.3914990425 0.9550000000 0.0011389256

651 2 windows install 0.3933839798 0.7530000000 0.0012989044

652 2 windows install 0.3916718960 0.8890000000 0.0009899139

653 2 windows install 0.1902680397 0.7270000000 0.0009901524

654 2 windows install 0.3929109573 0.6940000000 0.0012979507

655 2 windows install 0.1923339367 0.7230000000 0.0022950172

656 2 windows install 0.3908171654 0.9510000000 0.0007858276

657 2 windows install 0.1929228306 0.6890000000 0.0038139820

658 2 windows install 0.3888111115 0.9430000000 0.0012140274

659 2 windows install 0.3918228149 0.9050000000 0.0008628368

660 2 windows install 0.3887650967 0.9580000000 0.0011618137

661 2 windows install 0.3836030960 0.9720000000 0.0030860901

662 2 windows install 0.3904969692 0.9380000000 0.0003902912

663 2 windows install 0.4128618240 1.0300000000 0.0029747486

664 2 windows install 0.4168069363 0.9990000000 0.0008847713

Page 80: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLVI

665 2 windows install 0.3844978809 0.8250000000 0.0024130344

666 2 windows install 0.3933119774 1.0070000000 0.0013720989

667 2 windows install 0.4138820171 0.9240000000 0.0039329529

668 2 windows install 0.3828499317 1.0340000000 0.0032958984

669 2 windows install 0.1907939911 0.8890000000 0.0016429424

670 2 windows install 0.4254310131 1.1180000000 0.0045229123

671 2 windows install 0.4150550365 1.0550000000 0.0045239925

672 2 windows install 0.3888130188 0.9540000000 0.0012249947

673 2 windows install 0.3947780132 1.1160000000 0.0010809898

674 2 windows install 0.1917810440 0.7260000000 0.0024819374

675 2 windows install 0.4101171494 1.0210000000 0.0028440952

676 2 windows install 0.3923890591 1.0040000000 0.0006799698

677 2 windows install 0.4117350578 1.1230000000 0.0029339790

678 2 windows install 0.4266960621 1.1080000000 0.0025401142

679 2 windows install 0.4130938053 1.0620000000 0.0032401085

680 2 windows install 0.4138588905 1.1100000000 0.0034770966

681 2 windows install 0.4086129665 1.0550000000 0.0024101734

682 2 windows install 0.4089181423 0.7920000000 0.0024778843

683 2 windows install 0.3905420303 0.9620000000 0.0003991127

684 2 windows install 0.3928668499 1.0560000000 0.0011210442

685 2 windows install 0.3921990395 1.1380000000 0.0007112026

686 2 windows install 0.1869928837 0.7930000000 0.0015189648

687 2 windows install 0.1867048740 0.7450000000 0.0008547306

688 2 windows install 0.4030799866 0.9960000000 0.0005218983

689 2 windows install 0.3870851994 0.9400000000 0.0016279221

690 2 windows install 0.3902821541 0.9360000000 0.0004181862

691 2 windows install 0.3950059414 0.9730000000 0.0015108585

692 2 windows install 0.4156110287 0.9630000000 0.0044689178

693 2 windows install 0.3882520199 0.8980000000 0.0007240772

694 2 windows install 0.3908991814 0.9090000000 0.0008468628

695 2 windows install 0.1927039623 0.5430000000 0.0036697388

696 2 windows install 0.3968958855 0.9070000000 0.0007660389

697 2 windows install 0.3908610344 0.9120000000 0.0007958412

698 2 windows install 0.3965477943 0.9470000000 0.0006270409

699 2 windows install 0.3946099281 0.9270000000 0.0007522106

700 2 windows install 0.3970820904 0.8890000000 0.0009896755

701 2 windows install 0.3907430172 0.6920000000 0.0005650520

702 2 windows install 0.3890249729 0.9210000000 0.0012187958

703 2 windows install 0.4006369114 0.7560000000 0.0000572205

704 2 windows install 0.3905789852 0.7310000000 0.0004127026

705 2 windows install 0.3907780647 0.9720000000 0.0007040501

706 2 windows install 0.3982160091 0.9670000000 0.0004370213

707 2 windows install 0.3929910660 0.9370000000 0.0014219284

708 2 windows install 0.1929349899 0.7120000000 0.0038590431

709 2 windows install 0.3907668591 0.9020000000 0.0006759167

710 2 windows install 0.3861999512 0.9010000000 0.0019140244

711 2 windows install 0.3887338638 0.9050000000 0.0010190010

712 2 windows install 0.3921380043 0.8780000000 0.0007278919

Page 81: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLVII

713 2 windows install 0.1904470921 0.7220000000 0.0010561943

714 2 windows install 0.3907990456 0.7150000000 0.0007321835

715 2 windows install 0.1881420612 0.7220000000 0.0009539127

716 2 windows install 0.4083681107 0.9480000000 0.0025489330

717 2 windows install 0.4046680927 0.9660000000 0.0004801750

718 2 windows install 0.4092619419 0.8480000000 0.0030508041

719 2 windows install 0.3906800747 0.9110000000 0.0005080700

720 2 windows install 0.4085159302 0.9380000000 0.0023791790

721 2 windows install 0.1989548206 0.7230000000 0.1109099388

722 2 windows install 0.1945729256 0.7300000000 0.0053420067

723 2 windows install 0.4026689529 0.7880000000 0.0002839565

724 2 windows install 0.3950190544 1.0770000000 0.0014867783

725 2 windows install 0.3911161423 0.9160000000 0.0012040138

726 2 windows install 0.1848630905 0.7110000000 0.0010869503

727 2 windows install 0.3789231777 0.9020000000 0.0024459362

728 2 windows install 0.3942840099 0.9520000000 0.0006899834

729 2 windows install 0.3808240891 0.7210000000 0.0027370453

730 2 windows install 0.4071798325 1.0590000000 0.0022592545

731 2 windows install 0.3932218552 1.0200000000 0.0014069080

732 2 windows install 0.3907949924 0.8890000000 0.0007219315

733 2 windows install 0.1936678886 0.7000000000 0.0052268505

734 2 windows install 0.3948779106 0.9430000000 0.0015280247

735 2 windows install 0.3870420456 0.9620000000 0.0017156601

736 2 windows install 0.1961388588 0.8010000000 0.0110380650

737 2 windows install 0.4026749134 0.9520000000 0.0005209446

738 2 windows install 0.1976270676 0.7540000000 0.1093022823

739 2 windows install 0.3930768967 0.8940000000 0.0014290810

740 2 windows install 0.4128718376 0.9230000000 0.0027849674

741 2 windows install 0.3948981762 0.9630000000 0.0015611649

742 2 windows install 0.3950049877 0.9530000000 0.0015110970

743 2 windows install 0.3806819916 0.9160000000 0.0024228096

744 2 windows install 0.3967161179 0.9930000000 0.0007591248

745 2 windows install 0.3869221210 0.9130000000 0.0016829967

746 2 windows install 0.3915979862 0.9380000000 0.0010240078

747 2 windows install 0.3948850632 0.9040000000 0.0015270710

748 2 windows install 0.3906300068 0.9130000000 0.0004839897

749 2 windows install 0.1872529984 0.7270000000 0.0014219284

750 2 windows install 0.4101400375 0.9650000000 0.0028719902

751 2 windows install 0.3937649727 0.9180000000 0.0011792183

752 2 windows install 0.3912711143 0.9470000000 0.0012147427

753 2 windows install 0.3908500671 0.7570000000 0.0008099079

754 2 windows install 0.3960931301 0.8860000000 0.0007090569

755 2 windows install 0.3964140415 0.9490000000 0.0005900860

756 2 windows install 0.3968389034 0.9940000000 0.0007739067

757 2 windows install 0.3933629990 0.8280000000 0.0013649464

758 2 windows install 0.1986858845 0.7340000000 0.1093120575

759 2 windows install 0.3890030384 0.8850000000 0.0012381077

760 2 windows install 0.1905958652 0.6970000000 0.0012519360

Page 82: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLVIII

761 2 windows install 0.3887400627 0.9180000000 0.0011029243

762 2 windows install 0.3907480240 0.8790000000 0.0005707741

763 2 windows install 0.3885798454 0.6840000000 0.0007460117

764 2 windows install 0.3911108971 0.9100000000 0.0011930466

765 2 windows install 0.1948869228 0.6910000000 0.0081441402

766 2 windows install 0.1933588982 0.6930000000 0.0053081512

767 2 windows install 0.3924090862 0.9290000000 0.0006608963

768 2 windows install 0.3927111626 0.9660000000 0.0006821156

769 2 windows install 0.4105710983 0.9590000000 0.0021650791

770 2 windows install 0.4018640518 0.7330000000 0.0007848740

771 2 windows install 0.3987081051 0.9160000000 0.0004348755

772 2 windows install 0.3931438923 0.9240000000 0.0014171600

773 2 windows install 0.3948009014 0.6940000000 0.0013270378

774 2 windows install 0.3870539665 0.9520000000 0.0016093254

775 2 windows install 0.3846290112 0.9180000000 0.0025498867

776 2 windows install 0.3928170204 1.0290000000 0.0009300709

777 2 windows install 0.3922080994 0.9230000000 0.0007333755

778 2 windows install 0.3969888687 0.9220000000 0.0008139610

779 2 windows install 0.3876810074 0.7850000000 0.0010800362

780 2 windows install 0.4128139019 0.9580000000 0.0032486916

781 2 windows install 0.3899261951 1.0210000000 0.0006010532

782 2 windows install 0.1900629997 0.7150000000 0.0010650158

783 2 windows install 0.4048621655 0.7310000000 0.0005159378

784 2 windows install 0.3908510208 0.9220000000 0.0008080006

785 2 windows install 0.1888811588 0.7290000000 0.0017678738

786 2 windows install 0.3909671307 0.9030000000 0.0011010170

787 2 windows install 0.1908760071 0.5440000000 0.0018420219

788 2 windows install 0.3928599358 0.7260000000 0.0011141300

789 2 windows install 0.3904149532 0.9250000000 0.0003681183

790 2 windows install 0.1897640228 0.7380000000 0.0012152195

791 2 windows install 0.1882159710 0.7240000000 0.0010318756

792 2 windows install 0.3881299496 0.8960000000 0.0008819103

793 2 windows install 0.3933448792 0.9250000000 0.0013670921

794 2 windows install 0.3888740540 0.6890000000 0.0012369156

795 2 windows install 0.1905589104 0.7900000000 0.0010712147

796 2 windows install 0.3863930702 0.9480000000 0.0018770695

797 2 windows install 0.3919930458 0.9930000000 0.0008130074

798 2 windows install 0.3950729370 0.9160000000 0.0015058517

799 2 windows install 0.3881990910 0.7050000000 0.0008327961

800 2 windows install 0.3965749741 0.7410000000 0.0006251335

801 2 java get time 0.3985929489 0.7280000000 0.0004172325

802 2 java get time 0.3860709667 0.9380000000 0.0020833015

803 2 java get time 0.3855049610 1.0070000000 0.0022528172

804 2 java get time 0.4090640545 0.9320000000 0.0031819344

805 2 java get time 0.3936319351 0.9560000000 0.0012068748

806 2 java get time 0.3913381100 0.9940000000 0.0011880398

807 2 java get time 0.3895771503 1.0720000000 0.0009517670

808 2 java get time 0.1915059090 0.7350000000 0.0017080307

Page 83: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XLIX

809 2 java get time 0.3929560184 0.9350000000 0.0014548302

810 2 java get time 0.3905990124 1.0380000000 0.0004839897

811 2 java get time 0.4112989902 1.0990000000 0.0028572083

812 2 java get time 0.1955800056 0.5320000000 0.0102379322

813 2 java get time 0.4045720100 0.9910000000 0.0005650520

814 2 java get time 0.4009220600 0.9180000000 0.0000338554

815 2 java get time 0.3949749470 1.0840000000 0.0015070438

816 2 java get time 0.3974411488 1.0340000000 0.0008227825

817 2 java get time 0.3905968666 0.7430000000 0.0004892349

818 2 java get time 0.3911778927 1.0220000000 0.0012331009

819 2 java get time 0.3974161148 0.9530000000 0.0008680820

820 2 java get time 0.3892788887 1.0820000000 0.0011968613

821 2 java get time 0.1988620758 0.7490000000 0.1094701290

822 2 java get time 0.3886859417 0.7090000000 0.0010161400

823 2 java get time 0.1932768822 0.7820000000 0.0045781136

824 2 java get time 0.3937180042 1.0460000000 0.0011820793

825 2 java get time 0.4066078663 0.9180000000 0.0019140244

826 2 java get time 0.3989949226 1.0150000000 0.0006690025

827 2 java get time 0.3977150917 0.9810000000 0.0008950233

828 2 java get time 0.3927669525 0.9570000000 0.0007550716

829 2 java get time 0.4005398750 1.0570000000 0.0000259876

830 2 java get time 0.3987591267 0.9160000000 0.0007169247

831 2 java get time 0.3929722309 0.9320000000 0.0014579296

832 2 java get time 0.4131281376 1.0010000000 0.0034317970

833 2 java get time 0.3962359428 0.9220000000 0.0006949902

834 2 java get time 0.1921560764 0.5280000000 0.0023660660

835 2 java get time 0.3891592026 0.9330000000 0.0011761189

836 2 java get time 0.4025640488 0.9110000000 0.0007479191

837 2 java get time 0.3845188618 0.8970000000 0.0024750233

838 2 java get time 0.3866090775 0.8960000000 0.0019030571

839 2 java get time 0.1924459934 0.6960000000 0.0025000572

840 2 java get time 0.4105830193 0.9130000000 0.0022532940

841 2 java get time 0.3933749199 0.9910000000 0.0013418198

842 2 java get time 0.3956048489 0.9220000000 0.0011031628

843 2 java get time 0.3910930157 0.9520000000 0.0011999607

844 2 java get time 0.3858838081 0.9240000000 0.0020968914

845 2 java get time 0.3914561272 1.0120000000 0.0011429787

846 2 java get time 0.4001019001 0.9690000000 0.0002923012

847 2 java get time 0.4159049988 0.9460000000 0.0023100376

848 2 java get time 0.3965909481 0.9270000000 0.0005710125

849 2 java get time 0.3909358978 1.0430000000 0.0009949207

850 2 java get time 0.3953130245 0.9760000000 0.0013108253

851 2 java get time 0.4024598598 0.6980000000 0.0009362698

852 2 java get time 0.3966798782 0.9840000000 0.0007410049

853 2 java get time 0.3980460167 0.9670000000 0.0004341602

854 2 java get time 0.3917429447 0.9290000000 0.0009570122

855 2 java get time 0.4006941319 1.0260000000 0.0000607967

856 2 java get time 0.3972489834 0.9310000000 0.0009131432

Page 84: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

L

857 2 java get time 0.4049379826 0.9600000000 0.0005631447

858 2 java get time 0.3948748112 1.0640000000 0.0014688969

859 2 java get time 0.3969080448 0.9310000000 0.0007638931

860 2 java get time 0.3909180164 0.9320000000 0.0009570122

861 2 java get time 0.4046778679 0.9190000000 0.0005111694

862 2 java get time 0.3934779167 0.9210000000 0.0013058186

863 2 java get time 0.3825340271 0.9210000000 0.0027477741

864 2 java get time 0.1906650066 0.5190000000 0.0014019012

865 2 java get time 0.3993961811 0.9560000000 0.0004220009

866 2 java get time 0.3930618763 0.9520000000 0.0014390945

867 2 java get time 0.3910119534 0.9340000000 0.0011699200

868 2 java get time 0.1963200569 0.7070000000 0.1086790562

869 2 java get time 0.3983099461 0.9430000000 0.0004959106

870 2 java get time 0.3840751648 0.9230000000 0.0026531219

871 2 java get time 0.3920099735 0.9040000000 0.0008249283

872 2 java get time 0.3924450874 1.0290000000 0.0006189346

873 2 java get time 0.4161069393 0.9730000000 0.0021052361

874 2 java get time 0.4170188904 0.9930000000 0.0008337498

875 2 java get time 0.3931219578 1.0360000000 0.0014240742

876 2 java get time 0.4175801277 0.9690000000 0.0066058636

877 2 java get time 0.3913609982 0.6760000000 0.0011930466

878 2 java get time 0.3931639194 0.8830000000 0.0014028549

879 2 java get time 0.3832180500 0.8860000000 0.0031130314

880 2 java get time 0.1908378601 0.7510000000 0.0018060207

881 2 java get time 0.4035310745 0.9260000000 0.0008351803

882 2 java get time 0.1896040440 0.5120000000 0.0012199879

883 2 java get time 0.3928818703 1.0400000000 0.0012249947

884 2 java get time 0.3897008896 0.8970000000 0.0008227825

885 2 java get time 0.3893039227 0.8770000000 0.0011479855

886 2 java get time 0.3947069645 0.7340000000 0.0008039474

887 2 java get time 0.3967359066 0.9110000000 0.0007581711

888 2 java get time 0.3923680782 0.9190000000 0.0006620884

889 2 java get time 0.3859760761 0.9100000000 0.0020902157

890 2 java get time 0.3923509121 0.9290000000 0.0006759167

891 2 java get time 0.3895750046 0.9250000000 0.0009679794

892 2 java get time 0.3872170448 0.9870000000 0.0015101433

893 2 java get time 0.3824419975 0.9910000000 0.0025050640

894 2 java get time 0.4107711315 0.9460000000 0.0027160645

895 2 java get time 0.3946449757 0.9140000000 0.0007779598

896 2 java get time 0.4112858772 0.9580000000 0.0027260780

897 2 java get time 0.3911042213 1.0350000000 0.0011980534

898 2 java get time 0.3986079693 0.9080000000 0.0004119873

899 2 java get time 0.4120819569 0.9580000000 0.0032808781

900 2 java get time 0.3914890289 0.9060000000 0.0011410713

901 2 java get time 0.3905079365 0.9400000000 0.0003931522

902 2 java get time 0.3987278938 0.8670000000 0.0006010532

903 2 java get time 0.3948960304 0.7170000000 0.0015430450

904 2 java get time 0.3947360516 0.6900000000 0.0008778572

Page 85: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LI

905 2 java get time 0.3974058628 0.9320000000 0.0009586811

906 2 java get time 0.3932921886 0.9170000000 0.0013792515

907 2 java get time 0.3953521252 0.9380000000 0.0013120174

908 2 java get time 0.3813660145 0.6830000000 0.0022461414

909 2 java get time 0.4053399563 0.9120000000 0.0021300316

910 2 java get time 0.3869709969 0.9300000000 0.0016410351

911 2 java get time 0.4007759094 0.9200000000 0.0000169277

912 2 java get time 0.3934588432 0.9510000000 0.0013079643

913 2 java get time 0.2016398907 0.7570000000 0.1089899540

914 2 java get time 0.3932809830 0.9130000000 0.0013802052

915 2 java get time 0.3917558193 0.9220000000 0.0009369850

916 2 java get time 0.3814609051 0.9300000000 0.0022399426

917 2 java get time 0.3962309361 0.7250000000 0.0006549358

918 2 java get time 0.3948619366 0.7180000000 0.0014662743

919 2 java get time 0.3891999722 0.9620000000 0.0011849403

920 2 java get time 0.3951139450 0.9070000000 0.0015292168

921 2 java get time 0.3932418823 0.9150000000 0.0014078617

922 2 java get time 0.3936090469 1.0130000000 0.0012481213

923 2 java get time 0.3914160728 0.9120000000 0.0011444092

924 2 java get time 0.3906919956 0.8880000000 0.0005321503

925 2 java get time 0.3823199272 0.9440000000 0.0024271011

926 2 java get time 0.4044489861 0.9250000000 0.0006141663

927 2 java get time 0.3907990456 0.9180000000 0.0007441044

928 2 java get time 0.3960628510 0.9410000000 0.0008177757

929 2 java get time 0.1885349751 0.7550000000 0.0017223358

930 2 java get time 0.3893821239 0.9280000000 0.0009839535

931 2 java get time 0.3917369843 0.9150000000 0.0009770393

932 2 java get time 0.1908199787 0.7310000000 0.0017480850

933 2 java get time 0.3905911446 0.9070000000 0.0004818439

934 2 java get time 0.3942191601 1.0740000000 0.0007688999

935 2 java get time 0.1950709820 0.7540000000 0.0099701881

936 2 java get time 0.4021489620 0.9190000000 0.0008549690

937 2 java get time 0.3946700096 0.9950000000 0.0007658005

938 2 java get time 0.3862519264 0.7890000000 0.0018949509

939 2 java get time 0.3851861954 0.9410000000 0.0022819042

940 2 java get time 0.3916370869 0.9080000000 0.0009999275

941 2 java get time 0.3915071487 0.9820000000 0.0011398792

942 2 java get time 0.3986670971 0.9490000000 0.0003931522

943 2 java get time 0.3904151917 0.9090000000 0.0003650188

944 2 java get time 0.3959810734 0.9290000000 0.0008320808

945 2 java get time 0.3825249672 0.9870000000 0.0026681423

946 2 java get time 0.3907821178 0.9280000000 0.0007030964

947 2 java get time 0.3977179527 0.9100000000 0.0008058548

948 2 java get time 0.3957290649 0.8950000000 0.0009722710

949 2 java get time 0.3895950317 0.9210000000 0.0009047985

950 2 java get time 0.3892929554 0.9180000000 0.0011630058

951 2 java get time 0.3874828815 0.8800000000 0.0012459755

952 2 java get time 0.3991129398 0.9480000000 0.0004942417

Page 86: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LII

953 2 java get time 0.4169120789 0.9250000000 0.0003042221

954 2 java get time 0.1871399879 0.7550000000 0.0014300346

955 2 java get time 0.1930670738 0.7410000000 0.0039589405

956 2 java get time 0.3807671070 0.8960000000 0.0027930737

957 2 java get time 0.3947670460 1.0030000000 0.0009880066

958 2 java get time 0.3933179379 0.9060000000 0.0013749599

959 2 java get time 0.3973309994 0.9220000000 0.0008821487

960 2 java get time 0.3929758072 0.9350000000 0.0014548302

961 2 java get time 0.1962339878 0.7350000000 0.1073310375

962 2 java get time 0.3973948956 0.9250000000 0.0009398460

963 2 java get time 0.3945319653 0.8790000000 0.0006709099

964 2 java get time 0.3909981251 0.8780000000 0.0011548996

965 2 java get time 0.3958959579 0.9370000000 0.0008282661

966 2 java get time 0.3941688538 0.7500000000 0.0008208752

967 2 java get time 0.3944411278 0.9060000000 0.0006151199

968 2 java get time 0.3967108727 0.9390000000 0.0007290840

969 2 java get time 0.4105479717 0.9850000000 0.0023231506

970 2 java get time 0.3983819485 0.9480000000 0.0004897118

971 2 java get time 0.3855059147 0.9450000000 0.0022351742

972 2 java get time 0.1949751377 0.7420000000 0.0102300644

973 2 java get time 0.3975470066 0.9370000000 0.0008320808

974 2 java get time 0.1933269501 0.5130000000 0.0053801537

975 2 java get time 0.3946449757 0.9100000000 0.0007719994

976 2 java get time 0.3947570324 0.7380000000 0.0009450912

977 2 java get time 0.3917160034 0.9100000000 0.0009870529

978 2 java get time 0.1924018860 0.7440000000 0.0023939610

979 2 java get time 0.3927400112 0.9190000000 0.0007381439

980 2 java get time 0.3882739544 0.9050000000 0.0006921291

981 2 java get time 0.1920921803 0.5040000000 0.0024068356

982 2 java get time 0.1930940151 0.7060000000 0.0042760372

983 2 java get time 0.3970191479 1.0320000000 0.0008428097

984 2 java get time 0.3884730339 1.0050000000 0.0006940365

985 2 java get time 0.3939080238 0.9170000000 0.0009460449

986 2 java get time 0.4051599503 0.9570000000 0.0005359650

987 2 java get time 0.3934550285 0.9400000000 0.0012938976

988 2 java get time 0.3828399181 0.9240000000 0.0032548904

989 2 java get time 0.3944420815 0.9210000000 0.0006110668

990 2 java get time 0.1954870224 0.7160000000 0.0106220245

991 2 java get time 0.3834109306 0.9000000000 0.0031588078

992 2 java get time 0.3956720829 1.0640000000 0.0010962486

993 2 java get time 0.1917819977 0.5360000000 0.0022346973

994 2 java get time 0.1947050095 0.7020000000 0.0064308643

995 2 java get time 0.3921070099 0.9300000000 0.0007300377

996 2 java get time 0.1882231236 0.6910000000 0.0009748936

997 2 java get time 0.3926620483 0.8910000000 0.0006582737

998 2 java get time 0.3970651627 0.7400000000 0.0008609295

999 2 java get time 0.3946042061 1.1000000000 0.0006668568

1000 2 java get time 0.3955128193 0.9580000000 0.0011520386

Page 87: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LIII

1001 2 JavaScript Cookies 0.4027509689 0.8900000000 0.0005521774

1002 2 JavaScript Cookies 0.3936779499 0.9070000000 0.0011999607

1003 2 JavaScript Cookies 0.3935301304 0.7000000000 0.0012629032

1004 2 JavaScript Cookies 0.3879220486 1.0570000000 0.0010211468

1005 2 JavaScript Cookies 0.3912639618 0.9110000000 0.0012800694

1006 2 JavaScript Cookies 0.3977270126 0.7510000000 0.0007972717

1007 2 JavaScript Cookies 0.4001209736 0.9470000000 0.0003178120

1008 2 JavaScript Cookies 0.1909480095 0.8000000000 0.0018939972

1009 2 JavaScript Cookies 0.3963220119 0.9250000000 0.0006439686

1010 2 JavaScript Cookies 0.4086270332 0.9500000000 0.0024259090

1011 2 JavaScript Cookies 0.3908050060 0.9150000000 0.0007357597

1012 2 JavaScript Cookies 0.3954000473 1.1230000000 0.0012793541

1013 2 JavaScript Cookies 0.4052541256 0.9280000000 0.0017209053

1014 2 JavaScript Cookies 0.3958709240 1.0080000000 0.0008819103

1015 2 JavaScript Cookies 0.3919100761 0.9350000000 0.0007939339

1016 2 JavaScript Cookies 0.3978419304 0.9230000000 0.0007503033

1017 2 JavaScript Cookies 0.3980419636 1.0530000000 0.0004730225

1018 2 JavaScript Cookies 0.3965837955 0.9310000000 0.0005760193

1019 2 JavaScript Cookies 0.4041590691 1.0500000000 0.0010240078

1020 2 JavaScript Cookies 0.1899600029 0.9020000000 0.0011630058

1021 2 JavaScript Cookies 0.3904418945 0.7090000000 0.0003790855

1022 2 JavaScript Cookies 0.3968060017 0.8850000000 0.0008270741

1023 2 JavaScript Cookies 0.3960468769 0.9060000000 0.0008170605

1024 2 JavaScript Cookies 0.3940839767 0.9120000000 0.0008459091

1025 2 JavaScript Cookies 0.3902051449 0.9120000000 0.0004682541

1026 2 JavaScript Cookies 0.3918139935 0.9060000000 0.0009348392

1027 2 JavaScript Cookies 0.3989498615 0.9450000000 0.0007081032

1028 2 JavaScript Cookies 0.3935620785 0.9330000000 0.0012440681

1029 2 JavaScript Cookies 0.3935000896 1.0700000000 0.0012850761

1030 2 JavaScript Cookies 0.1874039173 0.8230000000 0.0015211105

1031 2 JavaScript Cookies 0.3926241398 0.9040000000 0.0006241798

1032 2 JavaScript Cookies 0.3863480091 0.8980000000 0.0018639565

1033 2 JavaScript Cookies 0.3908059597 0.9190000000 0.0007722378

1034 2 JavaScript Cookies 0.3950760365 0.9160000000 0.0015330315

1035 2 JavaScript Cookies 0.3953731060 0.9540000000 0.0012929440

1036 2 JavaScript Cookies 0.3947780132 0.8850000000 0.0011301041

1037 2 JavaScript Cookies 0.3983080387 0.8910000000 0.0004980564

1038 2 JavaScript Cookies 0.4002008438 0.9250000000 0.0002110004

1039 2 JavaScript Cookies 0.3993008137 0.9650000000 0.0006058216

1040 2 JavaScript Cookies 0.1925349236 0.6780000000 0.0027229786

1041 2 JavaScript Cookies 0.3865280151 0.8880000000 0.0019090176

1042 2 JavaScript Cookies 0.3910560608 0.9040000000 0.0011909008

1043 2 JavaScript Cookies 0.3926308155 0.8830000000 0.0006339550

1044 2 JavaScript Cookies 0.3896760941 0.9890000000 0.0008609295

1045 2 JavaScript Cookies 0.3827791214 0.7020000000 0.0032961369

1046 2 JavaScript Cookies 0.3906900883 0.9900000000 0.0005328655

1047 2 JavaScript Cookies 0.3920629025 0.9120000000 0.0007679462

1048 2 JavaScript Cookies 0.1934330463 0.7100000000 0.0054440498

Page 88: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LIV

1049 2 JavaScript Cookies 0.3932459354 0.9230000000 0.0013999939

1050 2 JavaScript Cookies 0.1923589706 0.7100000000 0.0023758411

1051 2 JavaScript Cookies 0.3897640705 0.9420000000 0.0006771088

1052 2 JavaScript Cookies 0.1970078945 0.8970000000 0.1103529930

1053 2 JavaScript Cookies 0.3963189125 0.9090000000 0.0007009506

1054 2 JavaScript Cookies 0.4008679390 0.7080000000 0.0000131130

1055 2 JavaScript Cookies 0.4011509418 0.9450000000 0.0004599094

1056 2 JavaScript Cookies 0.3803381920 0.9830000000 0.0023789406

1057 2 JavaScript Cookies 0.3967468739 1.0380000000 0.0007412434

1058 2 JavaScript Cookies 0.3976001740 0.9920000000 0.0008380413

1059 2 JavaScript Cookies 0.3942019939 0.9300000000 0.0008208752

1060 2 JavaScript Cookies 0.3987360001 0.8920000000 0.0006179810

1061 2 JavaScript Cookies 0.4104349613 0.9460000000 0.0028421879

1062 2 JavaScript Cookies 0.3910341263 1.0380000000 0.0011639595

1063 2 JavaScript Cookies 0.3946900368 0.9990000000 0.0007960796

1064 2 JavaScript Cookies 0.3951327801 1.0100000000 0.0013940334

1065 2 JavaScript Cookies 0.3920099735 0.6950000000 0.0008289814

1066 2 JavaScript Cookies 0.4077692032 1.0510000000 0.0027749538

1067 2 JavaScript Cookies 0.3966498375 0.9550000000 0.0006089211

1068 2 JavaScript Cookies 0.3963990211 0.8820000000 0.0006582737

1069 2 JavaScript Cookies 0.3935251236 0.9530000000 0.0012562275

1070 2 JavaScript Cookies 0.3906579018 0.9020000000 0.0004720688

1071 2 JavaScript Cookies 0.3928411007 0.9620000000 0.0010113716

1072 2 JavaScript Cookies 0.1938390732 0.5660000000 0.0049381256

1073 2 JavaScript Cookies 0.2004208565 0.7270000000 0.1098270416

1074 2 JavaScript Cookies 0.2027699947 0.8390000000 0.1071748734

1075 2 JavaScript Cookies 0.1885240078 0.8000000000 0.0017199516

1076 2 JavaScript Cookies 0.3927559853 0.8870000000 0.0007531643

1077 2 JavaScript Cookies 0.3857109547 0.7190000000 0.0021009445

1078 2 JavaScript Cookies 0.3999140263 0.9740000000 0.0003578663

1079 2 JavaScript Cookies 0.3984010220 0.6990000000 0.0004639626

1080 2 JavaScript Cookies 0.4073240757 0.9620000000 0.0027830601

1081 2 JavaScript Cookies 0.3954210281 0.7040000000 0.0012309551

1082 2 JavaScript Cookies 0.3908541203 1.0650000000 0.0008170605

1083 2 JavaScript Cookies 0.1898880005 0.5160000000 0.0011799335

1084 2 JavaScript Cookies 0.1903960705 0.7980000000 0.0010619164

1085 2 JavaScript Cookies 0.3856871128 0.7060000000 0.0021522045

1086 2 JavaScript Cookies 0.3847451210 0.9510000000 0.0025663376

1087 2 JavaScript Cookies 0.3898289204 0.9130000000 0.0006787777

1088 2 JavaScript Cookies 0.1865420341 0.7130000000 0.0004398823

1089 2 JavaScript Cookies 0.1935548782 0.7060000000 0.0053482056

1090 2 JavaScript Cookies 0.4014799595 0.7100000000 0.0006339550

1091 2 JavaScript Cookies 0.3978428841 0.9190000000 0.0006418228

1092 2 JavaScript Cookies 0.3990371227 0.9420000000 0.0005412102

1093 2 JavaScript Cookies 0.3942868710 0.9450000000 0.0006630421

1094 2 JavaScript Cookies 0.3954348564 0.9280000000 0.0012300014

1095 2 JavaScript Cookies 0.3925569057 0.7080000000 0.0006039143

1096 2 JavaScript Cookies 0.1948311329 0.7040000000 0.0068938732

Page 89: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LV

1097 2 JavaScript Cookies 0.3980221748 1.0540000000 0.0006399155

1098 2 JavaScript Cookies 0.3994081020 1.0200000000 0.0003852844

1099 2 JavaScript Cookies 0.3970770836 1.0200000000 0.0008800030

1100 2 JavaScript Cookies 0.3965220451 0.9200000000 0.0005798340

1101 2 JavaScript Cookies 0.1946010590 0.7130000000 0.0057380199

1102 2 JavaScript Cookies 0.3987739086 0.9020000000 0.0006728172

1103 2 JavaScript Cookies 0.4014310837 0.9520000000 0.0004351139

1104 2 JavaScript Cookies 0.3910391331 0.8920000000 0.0011687279

1105 2 JavaScript Cookies 0.3962121010 1.0370000000 0.0007131100

1106 2 JavaScript Cookies 0.1875078678 0.7040000000 0.0014216900

1107 2 JavaScript Cookies 0.3949100971 0.9540000000 0.0015220642

1108 2 JavaScript Cookies 0.3861007690 0.8870000000 0.0019571781

1109 2 JavaScript Cookies 0.4100749493 0.9310000000 0.0028669834

1110 2 JavaScript Cookies 0.4023029804 0.9240000000 0.0009069443

1111 2 JavaScript Cookies 0.3998620510 0.6960000000 0.0003070831

1112 2 JavaScript Cookies 0.3956999779 0.9240000000 0.0011069775

1113 2 JavaScript Cookies 0.3912110329 0.9120000000 0.0012528896

1114 2 JavaScript Cookies 0.4104900360 0.9550000000 0.0024402142

1115 2 JavaScript Cookies 0.3969609737 0.9270000000 0.0008547306

1116 2 JavaScript Cookies 0.3986229897 0.9940000000 0.0003850460

1117 2 JavaScript Cookies 0.3971059322 0.9240000000 0.0009729862

1118 2 JavaScript Cookies 0.3946821690 0.9200000000 0.0007600784

1119 2 JavaScript Cookies 0.3978221416 0.9200000000 0.0007972717

1120 2 JavaScript Cookies 0.3913669586 1.0300000000 0.0011868477

1121 2 JavaScript Cookies 0.3936800957 0.8980000000 0.0011780262

1122 2 JavaScript Cookies 0.3971111774 0.9170000000 0.0009777546

1123 2 JavaScript Cookies 0.1949160099 0.7680000000 0.0090537071

1124 2 JavaScript Cookies 0.3871967793 0.8970000000 0.0016279221

1125 2 JavaScript Cookies 0.3950891495 0.9290000000 0.0015339851

1126 2 JavaScript Cookies 0.3877019882 0.9100000000 0.0010590553

1127 2 JavaScript Cookies 0.4115772247 0.9290000000 0.0030829906

1128 2 JavaScript Cookies 0.3996090889 0.9110000000 0.0004820824

1129 2 JavaScript Cookies 0.3951280117 0.9090000000 0.0014209747

1130 2 JavaScript Cookies 0.3895559311 0.9110000000 0.0009708405

1131 2 JavaScript Cookies 0.3901381493 0.9090000000 0.0005028248

1132 2 JavaScript Cookies 0.3921649456 0.9230000000 0.0007238388

1133 2 JavaScript Cookies 0.1885638237 0.7190000000 0.0017070770

1134 2 JavaScript Cookies 0.3920331001 0.9260000000 0.0008120537

1135 2 JavaScript Cookies 0.3966519833 0.9390000000 0.0006301403

1136 2 JavaScript Cookies 0.1899800301 0.5250000000 0.0011069775

1137 2 JavaScript Cookies 0.1896789074 0.7180000000 0.0010960102

1138 2 JavaScript Cookies 0.3860640526 0.9320000000 0.0020852089

1139 2 JavaScript Cookies 0.3911368847 0.9250000000 0.0012221336

1140 2 JavaScript Cookies 0.3931350708 0.9190000000 0.0014138222

1141 2 JavaScript Cookies 0.1896049976 0.7880000000 0.0011489391

1142 2 JavaScript Cookies 0.3909299374 0.9460000000 0.0010011196

1143 2 JavaScript Cookies 0.1913139820 0.7270000000 0.0018830299

1144 2 JavaScript Cookies 0.3897199631 0.8950000000 0.0007650852

Page 90: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LVI

1145 2 JavaScript Cookies 0.1883440018 0.7010000000 0.0015919209

1146 2 JavaScript Cookies 0.3921809196 0.9930000000 0.0007169247

1147 2 JavaScript Cookies 0.1911499500 0.7070000000 0.0017008781

1148 2 JavaScript Cookies 0.3929469585 0.8880000000 0.0014379025

1149 2 JavaScript Cookies 0.3903319836 0.9570000000 0.0004210472

1150 2 JavaScript Cookies 0.3941700459 0.9190000000 0.0008127689

1151 2 JavaScript Cookies 0.3996767998 0.9160000000 0.0005660057

1152 2 JavaScript Cookies 0.4100759029 0.9180000000 0.0027551651

1153 2 JavaScript Cookies 0.3927259445 0.9590000000 0.0007369518

1154 2 JavaScript Cookies 0.4135880470 0.9600000000 0.0029029846

1155 2 JavaScript Cookies 0.4145760536 0.9990000000 0.0043988228

1156 2 JavaScript Cookies 0.4046101570 0.9380000000 0.0005271435

1157 2 JavaScript Cookies 0.3903510571 0.7420000000 0.0003709793

1158 2 JavaScript Cookies 0.4034171104 1.0290000000 0.0009520054

1159 2 JavaScript Cookies 0.1972689629 0.7030000000 0.1100959778

1160 2 JavaScript Cookies 0.3945360184 0.7280000000 0.0006322861

1161 2 JavaScript Cookies 0.3913810253 0.9140000000 0.0011560917

1162 2 JavaScript Cookies 0.3958530426 0.9070000000 0.0009498596

1163 2 JavaScript Cookies 0.2053208351 0.5160000000 0.0080628395

1164 2 JavaScript Cookies 0.3931610584 0.7010000000 0.0014092922

1165 2 JavaScript Cookies 0.3891561031 0.9210000000 0.0011570454

1166 2 JavaScript Cookies 0.4086029530 1.0210000000 0.0023341179

1167 2 JavaScript Cookies 0.3922188282 0.9090000000 0.0007719994

1168 2 JavaScript Cookies 0.3889141083 1.0070000000 0.0012540817

1169 2 JavaScript Cookies 0.3908970356 0.9130000000 0.0008308887

1170 2 JavaScript Cookies 0.3963398933 0.9450000000 0.0006949902

1171 2 JavaScript Cookies 0.3902409077 0.7070000000 0.0004518032

1172 2 JavaScript Cookies 0.3918490410 0.8980000000 0.0008578300

1173 2 JavaScript Cookies 0.1897938251 0.7290000000 0.0011909008

1174 2 JavaScript Cookies 0.3976149559 1.0220000000 0.0007548332

1175 2 JavaScript Cookies 0.3917880058 0.9300000000 0.0009341240

1176 2 JavaScript Cookies 0.3841481209 0.9130000000 0.0023941994

1177 2 JavaScript Cookies 0.3969371319 0.9260000000 0.0007541180

1178 2 JavaScript Cookies 0.3818469048 0.7510000000 0.0022180080

1179 2 JavaScript Cookies 0.3893589973 0.9310000000 0.0011377335

1180 2 JavaScript Cookies 0.1885139942 0.5200000000 0.0017259121

1181 2 JavaScript Cookies 0.1897611618 0.7170000000 0.0011301041

1182 2 JavaScript Cookies 0.3904929161 0.9680000000 0.0003900528

1183 2 JavaScript Cookies 0.3935620785 1.1700000000 0.0012512207

1184 2 JavaScript Cookies 0.4033160210 0.9540000000 0.0004780293

1185 2 JavaScript Cookies 0.4033029079 0.9270000000 0.0005371571

1186 2 JavaScript Cookies 0.3947770596 0.9160000000 0.0010011196

1187 2 JavaScript Cookies 0.4030189514 0.9160000000 0.0006437302

1188 2 JavaScript Cookies 0.1927371025 0.7510000000 0.0036728382

1189 2 JavaScript Cookies 0.3926868439 0.7090000000 0.0006668568

1190 2 JavaScript Cookies 0.3940348625 0.9110000000 0.0009160042

1191 2 JavaScript Cookies 0.3922169209 0.9350000000 0.0007562637

1192 2 JavaScript Cookies 0.3937439919 0.9210000000 0.0011658669

Page 91: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LVII

1193 2 JavaScript Cookies 0.3964519501 0.9230000000 0.0005793571

1194 2 JavaScript Cookies 0.3917009830 0.9250000000 0.0009829998

1195 2 JavaScript Cookies 0.3923399448 0.9330000000 0.0006721020

1196 2 JavaScript Cookies 0.3836371899 0.8900000000 0.0030748844

1197 2 JavaScript Cookies 0.3911058903 0.9290000000 0.0011937618

1198 2 JavaScript Cookies 0.4022090435 1.0090000000 0.0008187294

1199 2 JavaScript Cookies 0.3956010342 1.0000000000 0.0011372566

1200 2 JavaScript Cookies 0.3998401165 1.0030000000 0.0004711151

1201 3 windows install 0.4189429283 1.2330000000 0.0035998821

1202 3 windows install 0.1565060616 0.7050000000 0.0000197887

1203 3 windows install 0.3575060368 0.9000000000 0.0005991459

1204 3 windows install 0.3532159328 0.9410000000 0.0002238750

1205 3 windows install 0.3552749157 0.8960000000 0.0004889965

1206 3 windows install 0.1527600288 0.6900000000 0.0000522137

1207 3 windows install 0.3790178299 0.8960000000 0.0023272038

1208 3 windows install 0.3511800766 0.8600000000 0.0001299381

1209 3 windows install 0.3605830669 0.9230000000 0.0005478859

1210 3 windows install 0.3463411331 0.8720000000 0.0003688335

1211 3 windows install 0.3532159328 0.8860000000 0.0002121925

1212 3 windows install 0.3584499359 0.8790000000 0.0009901524

1213 3 windows install 0.3624720573 0.9500000000 0.0020062923

1214 3 windows install 0.3545358181 0.8820000000 0.0008018017

1215 3 windows install 0.3525378704 0.8740000000 0.0001449585

1216 3 windows install 0.3531050682 0.9240000000 0.0001499653

1217 3 windows install 0.3551399708 0.6730000000 0.0004599094

1218 3 windows install 0.3578920364 0.8990000000 0.0006558895

1219 3 windows install 0.3499619961 0.8660000000 0.0002808571

1220 3 windows install 0.3542368412 0.8790000000 0.0006990433

1221 3 windows install 0.3452749252 0.8940000000 0.0006840229

1222 3 windows install 0.1539680958 0.5070000000 0.0002720356

1223 3 windows install 0.3559010029 0.9780000000 0.0009031296

1224 3 windows install 0.3501901627 0.6490000000 0.0000522137

1225 3 windows install 0.3562550545 0.8490000000 0.0008080006

1226 3 windows install 0.3518152237 0.7410000000 0.0004911423

1227 3 windows install 0.1592979431 0.7580000000 0.0011520386

1228 3 windows install 0.3529160023 0.8620000000 0.0001969337

1229 3 windows install 0.3727340698 1.0260000000 0.0073339939

1230 3 windows install 0.3528769016 0.6660000000 0.0001959801

1231 3 windows install 0.3595540524 0.9800000000 0.0000360012

1232 3 windows install 0.3669259548 0.8750000000 0.0051658154

1233 3 windows install 0.3571228981 0.7960000000 0.0003418922

1234 3 windows install 0.3557991982 0.8670000000 0.0007212162

1235 3 windows install 0.3546650410 0.8640000000 0.0006871223

1236 3 windows install 0.3533978462 0.6970000000 0.0002839565

1237 3 windows install 0.3516521454 0.9600000000 0.0002961159

1238 3 windows install 0.3496210575 0.6740000000 0.0002419949

1239 3 windows install 0.3460888863 0.6410000000 0.0005357265

1240 3 windows install 0.3591668606 0.8890000000 0.0002100468

Page 92: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LVIII

1241 3 windows install 0.3595190048 0.8550000000 0.0000419617

1242 3 windows install 0.1581628323 0.7100000000 0.0010650158

1243 3 windows install 0.3550410271 0.8800000000 0.0004019737

1244 3 windows install 0.3531339169 0.6530000000 0.0001571178

1245 3 windows install 0.3561718464 0.8920000000 0.0007691383

1246 3 windows install 0.3535149097 0.9390000000 0.0004479885

1247 3 windows install 0.3726701736 1.0460000000 0.0093321800

1248 3 windows install 0.3550209999 0.7790000000 0.0003819466

1249 3 windows install 0.3564450741 0.6880000000 0.0004742146

1250 3 windows install 0.3554711342 0.8870000000 0.0006139278

1251 3 windows install 0.3503770828 0.6600000000 0.0000801086

1252 3 windows install 0.3483440876 0.6450000000 0.0001001358

1253 3 windows install 0.3571970463 0.8900000000 0.0004000664

1254 3 windows install 0.3503129482 0.8740000000 0.0000641346

1255 3 windows install 0.3567378521 0.8690000000 0.0003242493

1256 3 windows install 0.3548109531 0.8250000000 0.0004410744

1257 3 windows install 0.3591480255 0.8990000000 0.0001556873

1258 3 windows install 0.3425428867 0.8750000000 0.0010790825

1259 3 windows install 0.1507918835 0.7130000000 0.0004730225

1260 3 windows install 0.3546121120 0.7140000000 0.0007522106

1261 3 windows install 0.1536319256 0.7310000000 0.0004169941

1262 3 windows install 0.3670690060 0.9120000000 0.0049209595

1263 3 windows install 0.3516781330 0.7130000000 0.0003080368

1264 3 windows install 0.1499409676 0.6360000000 0.0006828308

1265 3 windows install 0.3688468933 0.8830000000 0.0099008083

1266 3 windows install 0.3575789928 0.8690000000 0.0006830692

1267 3 windows install 0.1538388729 0.6800000000 0.0005121231

1268 3 windows install 0.3648900986 0.8940000000 0.0034651756

1269 3 windows install 0.3588271141 0.8770000000 0.0000600815

1270 3 windows install 0.3673760891 0.8840000000 0.0048558712

1271 3 windows install 0.3549368382 0.8940000000 0.0004200935

1272 3 windows install 0.3604550362 0.8580000000 0.0001468658

1273 3 windows install 0.3524849415 0.8630000000 0.0002310276

1274 3 windows install 0.3471138477 0.8870000000 0.0000782013

1275 3 windows install 0.3531239033 0.8930000000 0.0001549721

1276 3 windows install 0.1501920223 0.6670000000 0.0006349087

1277 3 windows install 0.1638660431 0.7040000000 0.0023839474

1278 3 windows install 0.3500509262 0.8750000000 0.0000090599

1279 3 windows install 0.3685438633 0.9260000000 0.0092132092

1280 3 windows install 0.3547179699 0.9650000000 0.0005831718

1281 3 windows install 0.1684551239 0.6840000000 0.0149328709

1282 3 windows install 0.3506481647 0.6390000000 0.0001461506

1283 3 windows install 0.3425660133 0.8700000000 0.0015461445

1284 3 windows install 0.1532690525 0.4630000000 0.0000610352

1285 3 windows install 0.3491621017 0.9840000000 0.0000112057

1286 3 windows install 0.3555371761 0.9120000000 0.0006101131

1287 3 windows install 0.3604350090 0.6910000000 0.0002410412

1288 3 windows install 0.3527650833 0.8760000000 0.0001640320

Page 93: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LIX

1289 3 windows install 0.1531960964 0.6830000000 0.0000481606

1290 3 windows install 0.3529829979 0.8680000000 0.0002040863

1291 3 windows install 0.3505458832 0.8650000000 0.0000720024

1292 3 windows install 0.3538460732 0.7590000000 0.0006010532

1293 3 windows install 0.3530700207 0.8970000000 0.0001368523

1294 3 windows install 0.3598580360 0.8790000000 0.0001821518

1295 3 windows install 0.3599109650 0.6660000000 0.0002021790

1296 3 windows install 0.3568081856 0.8790000000 0.0002958775

1297 3 windows install 0.3574590683 0.8470000000 0.0005562305

1298 3 windows install 0.3531479836 0.8600000000 0.0002281666

1299 3 windows install 0.3514599800 0.8680000000 0.0001730919

1300 3 windows install 0.3599908352 0.7740000000 0.0001497269

1301 3 windows install 0.3549790382 0.8890000000 0.0004477501

1302 3 windows install 0.1505918503 0.7400000000 0.0004591942

1303 3 windows install 0.3568260670 0.8950000000 0.0002248287

1304 3 windows install 0.3769009113 0.8960000000 0.0042870045

1305 3 windows install 0.3604390621 0.9160000000 0.0002260208

1306 3 windows install 0.3525619507 0.8620000000 0.0001399517

1307 3 windows install 0.3478989601 0.8810000000 0.0002260208

1308 3 windows install 0.3556199074 0.8750000000 0.0006461143

1309 3 windows install 0.3705470562 0.9420000000 0.0091218948

1310 3 windows install 0.3543851376 0.8750000000 0.0007898808

1311 3 windows install 0.3549928665 0.9120000000 0.0004532337

1312 3 windows install 0.3507339954 0.8500000000 0.0000758171

1313 3 windows install 0.3531360626 0.8850000000 0.0001680851

1314 3 windows install 0.3532130718 0.8640000000 0.0002300739

1315 3 windows install 0.3510651588 0.8630000000 0.0001900196

1316 3 windows install 0.3539359570 0.8610000000 0.0006139278

1317 3 windows install 0.3519580364 0.9920000000 0.0005147457

1318 3 windows install 0.3626611233 0.8740000000 0.0022542477

1319 3 windows install 0.1697399616 0.7740000000 0.0038287640

1320 3 windows install 0.3547999859 0.8750000000 0.0004329681

1321 3 windows install 0.3553020954 0.7800000000 0.0005631447

1322 3 windows install 0.3591578007 0.8900000000 0.0000698566

1323 3 windows install 0.3503081799 0.8420000000 0.0000479221

1324 3 windows install 0.3476049900 0.8690000000 0.0001800060

1325 3 windows install 0.3494188786 0.9010000000 0.0000958443

1326 3 windows install 0.1504509449 0.8140000000 0.0005660057

1327 3 windows install 0.3489758968 0.9070000000 0.0000751019

1328 3 windows install 0.3565399647 0.8660000000 0.0003979206

1329 3 windows install 0.3556830883 0.8770000000 0.0006709099

1330 3 windows install 0.3547480106 0.6420000000 0.0004279613

1331 3 windows install 0.3490350246 0.8920000000 0.0000422001

1332 3 windows install 0.3506269455 0.8760000000 0.0000741482

1333 3 windows install 0.1487939358 0.6460000000 0.0017900467

1334 3 windows install 0.3560841084 0.8990000000 0.0008783340

1335 3 windows install 0.3555440903 0.9070000000 0.0006198883

1336 3 windows install 0.3607559204 0.8900000000 0.0005548000

Page 94: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LX

1337 3 windows install 0.3589599133 0.9530000000 0.0000960827

1338 3 windows install 0.3609058857 0.8800000000 0.0007870197

1339 3 windows install 0.3594939709 0.8730000000 0.0000560284

1340 3 windows install 0.3572270870 1.0410000000 0.0005102158

1341 3 windows install 0.3547220230 0.8790000000 0.0004379749

1342 3 windows install 0.3505039215 0.8590000000 0.0000212193

1343 3 windows install 0.3508517742 0.8840000000 0.0000929832

1344 3 windows install 0.3547658920 0.8450000000 0.0004010201

1345 3 windows install 0.3544809818 0.8910000000 0.0007817745

1346 3 windows install 0.1589751244 0.5100000000 0.0011432171

1347 3 windows install 0.3542070389 0.6660000000 0.0007157326

1348 3 windows install 0.3568120003 0.8840000000 0.0002400875

1349 3 windows install 0.3537800312 0.8870000000 0.0005040169

1350 3 windows install 0.3536748886 0.8660000000 0.0005538464

1351 3 windows install 0.1592547894 0.6980000000 0.0011911392

1352 3 windows install 0.3612749577 0.7040000000 0.0015769005

1353 3 windows install 0.3570349216 0.8550000000 0.0003700256

1354 3 windows install 0.3581700325 0.9260000000 0.0009441376

1355 3 windows install 0.3625500202 0.8730000000 0.0020899773

1356 3 windows install 0.3526031971 0.9590000000 0.0000700951

1357 3 windows install 0.3555371761 0.9180000000 0.0006089211

1358 3 windows install 0.3632099628 0.9040000000 0.0023930073

1359 3 windows install 0.1548230648 0.4570000000 0.0006389618

1360 3 windows install 0.3560109138 0.8630000000 0.0009200573

1361 3 windows install 0.3516740799 0.8870000000 0.0002973080

1362 3 windows install 0.3615880013 0.8790000000 0.0017600060

1363 3 windows install 0.3467578888 0.9350000000 0.0003030300

1364 3 windows install 0.3604240417 0.8810000000 0.0002129078

1365 3 windows install 0.3568818569 0.9890000000 0.0002830029

1366 3 windows install 0.3647789955 0.8570000000 0.0035238266

1367 3 windows install 0.1547100544 0.7120000000 0.0005319118

1368 3 windows install 0.3544640541 0.8720000000 0.0007770061

1369 3 windows install 0.1536509991 0.6860000000 0.0004830360

1370 3 windows install 0.3522610664 0.9080000000 0.0002760887

1371 3 windows install 0.3511490822 0.8580000000 0.0002181530

1372 3 windows install 0.3511149883 0.8940000000 0.0001790524

1373 3 windows install 0.1543920040 0.7090000000 0.0002679825

1374 3 windows install 0.3530509472 0.9030000000 0.0001881123

1375 3 windows install 0.3689930439 0.8780000000 0.0092329979

1376 3 windows install 0.3614251614 0.8850000000 0.0015668869

1377 3 windows install 0.3555669785 0.9450000000 0.0006666183

1378 3 windows install 0.3529939651 0.6830000000 0.0002088547

1379 3 windows install 0.3515920639 0.9110000000 0.0003020763

1380 3 windows install 0.1534981728 0.6710000000 0.0002002716

1381 3 windows install 0.3562860489 0.6650000000 0.0007970333

1382 3 windows install 0.3899519444 0.9160000000 0.0006139278

1383 3 windows install 0.3563928604 0.9240000000 0.0006411076

1384 3 windows install 0.3783710003 0.9230000000 0.0037629604

Page 95: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXI

1385 3 windows install 0.3465731144 1.0050000000 0.0003230572

1386 3 windows install 0.3537700176 0.8720000000 0.0005400181

1387 3 windows install 0.3686029911 1.0060000000 0.0095858574

1388 3 windows install 0.3751480579 0.9230000000 0.0042529106

1389 3 windows install 0.3682141304 0.7360000000 0.0058526993

1390 3 windows install 0.1518461704 0.6630000000 0.0005190372

1391 3 windows install 0.3547649384 0.8690000000 0.0004079342

1392 3 windows install 0.3509180546 0.6960000000 0.0001859665

1393 3 windows install 0.3528280258 0.9020000000 0.0001552105

1394 3 windows install 0.3642690182 0.9330000000 0.0019450188

1395 3 windows install 0.3519937992 0.8760000000 0.0005800724

1396 3 windows install 0.3661110401 0.8620000000 0.0040519238

1397 3 windows install 0.3617401123 0.8760000000 0.0015769005

1398 3 windows install 0.3544499874 0.8710000000 0.0007712841

1399 3 windows install 0.3505301476 0.8490000000 0.0000209808

1400 3 windows install 0.3514721394 0.6370000000 0.0002307892

1401 3 java get time 0.1991779804 0.9650000000 0.1099190712

1402 3 java get time 0.3534719944 0.9070000000 0.0003559589

1403 3 java get time 0.3603410721 0.9110000000 0.0002582073

1404 3 java get time 0.1510779858 0.6700000000 0.0007109642

1405 3 java get time 0.3553800583 0.9050000000 0.0005688667

1406 3 java get time 0.3465261459 0.8810000000 0.0004909039

1407 3 java get time 0.3610489368 0.9000000000 0.0013210773

1408 3 java get time 0.1527240276 0.6680000000 0.0001552105

1409 3 java get time 0.3483638763 0.8550000000 0.0001270771

1410 3 java get time 0.3546380997 0.7670000000 0.0007100105

1411 3 java get time 0.1580989361 0.5420000000 0.0013642311

1412 3 java get time 0.3548099995 0.8810000000 0.0004189014

1413 3 java get time 0.1570198536 0.4930000000 0.0007541180

1414 3 java get time 0.3516240120 0.6690000000 0.0002810955

1415 3 java get time 0.3531250954 0.9360000000 0.0001630783

1416 3 java get time 0.3530800343 0.8960000000 0.0001502037

1417 3 java get time 0.3540639877 0.8870000000 0.0006628036

1418 3 java get time 0.3550679684 0.9610000000 0.0004067421

1419 3 java get time 0.3510560989 0.8920000000 0.0001530647

1420 3 java get time 0.3605749607 0.8760000000 0.0003550053

1421 3 java get time 0.3540570736 0.8930000000 0.0006668568

1422 3 java get time 0.3528151512 0.9390000000 0.0001671314

1423 3 java get time 0.3508701324 0.9800000000 0.0001161098

1424 3 java get time 0.3573451042 0.9240000000 0.0005009174

1425 3 java get time 0.3561518192 0.8790000000 0.0008120537

1426 3 java get time 0.3528261185 0.6420000000 0.0001521111

1427 3 java get time 0.1507167816 0.6780000000 0.0005300045

1428 3 java get time 0.3579459190 0.8920000000 0.0008029938

1429 3 java get time 0.3530600071 0.8630000000 0.0001380444

1430 3 java get time 0.3586080074 0.8760000000 0.0005741119

1431 3 java get time 0.1525771618 0.6830000000 0.0001528263

1432 3 java get time 0.3534190655 0.8860000000 0.0003168583

Page 96: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXII

1433 3 java get time 0.1538388729 0.7920000000 0.0003948212

1434 3 java get time 0.3521938324 0.8770000000 0.0005669594

1435 3 java get time 0.3430559635 0.8410000000 0.0015017986

1436 3 java get time 0.3544571400 0.6660000000 0.0007641315

1437 3 java get time 0.3475370407 0.8750000000 0.0001339912

1438 3 java get time 0.3682971001 0.8950000000 0.0071141720

1439 3 java get time 0.3650288582 0.9190000000 0.0037286282

1440 3 java get time 0.3551478386 0.8930000000 0.0004298687

1441 3 java get time 0.3547520638 0.8980000000 0.0004317760

1442 3 java get time 0.3525300026 0.8640000000 0.0001518726

1443 3 java get time 0.3533098698 0.9680000000 0.0002770424

1444 3 java get time 0.1549119949 0.6830000000 0.0008182526

1445 3 java get time 0.3626480103 0.9120000000 0.0022809505

1446 3 java get time 0.3564200401 0.8700000000 0.0003840923

1447 3 java get time 0.3566229343 0.8940000000 0.0003900528

1448 3 java get time 0.3560299873 0.8720000000 0.0009131432

1449 3 java get time 0.3495419025 0.8900000000 0.0001609325

1450 3 java get time 0.3566980362 0.9310000000 0.0003521442

1451 3 java get time 0.3561930656 0.7100000000 0.0007901192

1452 3 java get time 0.3569660187 0.8690000000 0.0003478527

1453 3 java get time 0.3653678894 0.8710000000 0.0041098595

1454 3 java get time 0.1480879784 0.4950000000 0.0018730164

1455 3 java get time 0.3576641083 0.9230000000 0.0007369518

1456 3 java get time 0.3547739983 0.8620000000 0.0004270077

1457 3 java get time 0.3552548885 0.8740000000 0.0005011559

1458 3 java get time 0.3560290337 0.8910000000 0.0008869171

1459 3 java get time 0.3547708988 0.9040000000 0.0004031658

1460 3 java get time 0.3663699627 0.8890000000 0.0041389465

1461 3 java get time 0.3567268848 0.8900000000 0.0002748966

1462 3 java get time 0.1511969566 0.6900000000 0.0008392334

1463 3 java get time 0.1595070362 0.6770000000 0.0047719479

1464 3 java get time 0.1530029774 0.4420000000 0.0000147820

1465 3 java get time 0.1507267952 0.6590000000 0.0004851818

1466 3 java get time 0.3547139168 0.8960000000 0.0006039143

1467 3 java get time 0.1492629051 0.6620000000 0.0012528896

1468 3 java get time 0.3446991444 0.8390000000 0.0002927780

1469 3 java get time 0.3646128178 1.1040000000 0.0029199123

1470 3 java get time 0.3530561924 0.8930000000 0.0001640320

1471 3 java get time 0.3617119789 0.8740000000 0.0018451214

1472 3 java get time 0.3537499905 0.8510000000 0.0005590916

1473 3 java get time 0.3501701355 0.8900000000 0.0000429153

1474 3 java get time 0.3525269032 0.8590000000 0.0001490116

1475 3 java get time 0.3628139496 0.6810000000 0.0023248196

1476 3 java get time 0.3508191109 1.0020000000 0.0001118183

1477 3 java get time 0.3650631905 0.8750000000 0.0038411617

1478 3 java get time 0.3545720577 0.8470000000 0.0007667542

1479 3 java get time 0.3591208458 0.8510000000 0.0001709461

1480 3 java get time 0.3515357971 0.8880000000 0.0002770424

Page 97: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXIII

1481 3 java get time 0.3621909618 0.8850000000 0.0020170212

1482 3 java get time 0.3612561226 0.8980000000 0.0014250278

1483 3 java get time 0.3595321178 0.6690000000 0.0000100136

1484 3 java get time 0.3508348465 1.0690000000 0.0000967979

1485 3 java get time 0.3454589844 0.8730000000 0.0011031628

1486 3 java get time 0.3577830791 0.8580000000 0.0006906986

1487 3 java get time 0.1530618668 0.6770000000 0.0000209808

1488 3 java get time 0.3563549519 0.8690000000 0.0007579327

1489 3 java get time 0.3545680046 1.0810000000 0.0007631779

1490 3 java get time 0.3486771584 0.8730000000 0.0001118183

1491 3 java get time 0.3548390865 0.8590000000 0.0004348755

1492 3 java get time 0.3679950237 0.9680000000 0.0058650970

1493 3 java get time 0.3601191044 0.8880000000 0.0001740456

1494 3 java get time 0.3603341579 0.8730000000 0.0002529621

1495 3 java get time 0.3611891270 0.8930000000 0.0013518333

1496 3 java get time 0.3750419617 0.8840000000 0.0055651665

1497 3 java get time 0.3490140438 0.8510000000 0.0000469685

1498 3 java get time 0.3845479488 0.9630000000 0.0025141239

1499 3 java get time 0.3530769348 0.8790000000 0.0001349449

1500 3 java get time 0.3587260246 1.0200000000 0.0002768040

1501 3 java get time 0.3514571190 0.8920000000 0.0002348423

1502 3 java get time 0.3513250351 0.8500000000 0.0001940727

1503 3 java get time 0.3552651405 0.8490000000 0.0004789829

1504 3 java get time 0.3550980091 0.8600000000 0.0004608631

1505 3 java get time 0.3553919792 0.9660000000 0.0005819798

1506 3 java get time 0.3539969921 0.8950000000 0.0006229877

1507 3 java get time 0.3603849411 0.8930000000 0.0002200603

1508 3 java get time 0.3622729778 0.8920000000 0.0019099712

1509 3 java get time 0.3550138474 0.8690000000 0.0003890991

1510 3 java get time 0.3522179127 0.7090000000 0.0005657673

1511 3 java get time 0.3529758453 1.0140000000 0.0002009869

1512 3 java get time 0.3522379398 0.9230000000 0.0005710125

1513 3 java get time 0.3532450199 0.9980000000 0.0002038479

1514 3 java get time 0.1522200108 0.7290000000 0.0000789165

1515 3 java get time 0.3540740013 0.8710000000 0.0006680489

1516 3 java get time 0.3470208645 0.8880000000 0.0003082752

1517 3 java get time 0.3495309353 0.9540000000 0.0001029968

1518 3 java get time 0.3527920246 0.8800000000 0.0001668930

1519 3 java get time 0.3615128994 0.8280000000 0.0017991066

1520 3 java get time 0.3478460312 0.8730000000 0.0001099110

1521 3 java get time 0.1515870094 0.6840000000 0.0005519390

1522 3 java get time 0.3550059795 0.8910000000 0.0004088879

1523 3 java get time 0.3656339645 0.7480000000 0.0043020248

1524 3 java get time 0.3507850170 0.8430000000 0.0000979900

1525 3 java get time 0.3590600491 0.8900000000 0.0001769066

1526 3 java get time 0.3593499660 0.8790000000 0.0001752377

1527 3 java get time 0.3541009426 0.8380000000 0.0006809235

1528 3 java get time 0.3570580482 0.8930000000 0.0003919601

Page 98: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXIV

1529 3 java get time 0.1508359909 0.7030000000 0.0004410744

1530 3 java get time 0.3547320366 0.8670000000 0.0004289150

1531 3 java get time 0.3526210785 0.8560000000 0.0000610352

1532 3 java get time 0.3552401066 0.8890000000 0.0005059242

1533 3 java get time 0.3512351513 0.8860000000 0.0001509190

1534 3 java get time 0.1532809734 0.6780000000 0.0002510548

1535 3 java get time 0.3561270237 0.8760000000 0.0008368492

1536 3 java get time 0.3578188419 0.6730000000 0.0007097721

1537 3 java get time 0.3555040359 0.8920000000 0.0006110668

1538 3 java get time 0.3546180725 0.7560000000 0.0007669926

1539 3 java get time 0.3649878502 0.9250000000 0.0034732819

1540 3 java get time 0.3549239635 0.8950000000 0.0004189014

1541 3 java get time 0.3507211208 0.8370000000 0.0001478195

1542 3 java get time 0.3541340828 0.8580000000 0.0006699562

1543 3 java get time 0.3528690338 0.9410000000 0.0001597404

1544 3 java get time 0.3557229042 0.8930000000 0.0006670952

1545 3 java get time 0.3562259674 0.9140000000 0.0007770061

1546 3 java get time 0.3550050259 0.8600000000 0.0004529953

1547 3 java get time 0.1554181576 0.6710000000 0.0002119541

1548 3 java get time 0.3480288982 0.9030000000 0.0000071526

1549 3 java get time 0.3497450352 0.9380000000 0.0003070831

1550 3 java get time 0.3561358452 0.8850000000 0.0008127689

1551 3 java get time 0.3588349819 0.8290000000 0.0000810623

1552 3 java get time 0.3522820473 0.8850000000 0.0003149509

1553 3 java get time 0.3545758724 0.9060000000 0.0007901192

1554 3 java get time 0.3532569408 0.8310000000 0.0002281666

1555 3 java get time 0.3548641205 0.8580000000 0.0004122257

1556 3 java get time 0.3578839302 0.6910000000 0.0006601810

1557 3 java get time 0.3628602028 0.9040000000 0.0023839474

1558 3 java get time 0.3606281281 0.8630000000 0.0005469322

1559 3 java get time 0.3529510498 0.8710000000 0.0002250671

1560 3 java get time 0.3536319733 0.8850000000 0.0004849434

1561 3 java get time 0.3542370796 0.8750000000 0.0007078648

1562 3 java get time 0.1505498886 0.7080000000 0.0005810261

1563 3 java get time 0.3657100201 0.6740000000 0.0043790340

1564 3 java get time 0.3469219208 1.0450000000 0.0002100468

1565 3 java get time 0.3558061123 0.9580000000 0.0007591248

1566 3 java get time 0.3712990284 0.6930000000 0.0089511871

1567 3 java get time 0.3544988632 0.8760000000 0.0007748604

1568 3 java get time 0.1519920826 0.8110000000 0.0003080368

1569 3 java get time 0.3543810844 0.9040000000 0.0007660389

1570 3 java get time 0.3605489731 0.8620000000 0.0003299713

1571 3 java get time 0.3490099907 0.6940000000 0.0000491142

1572 3 java get time 0.1594491005 0.6980000000 0.0039949417

1573 3 java get time 0.3530199528 1.0040000000 0.0002119541

1574 3 java get time 0.1543040276 0.6780000000 0.0002210140

1575 3 java get time 0.3667919636 0.8520000000 0.0050220490

1576 3 java get time 0.1553170681 0.6570000000 0.0010030270

Page 99: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXV

1577 3 java get time 0.3571798801 0.8460000000 0.0003750324

1578 3 java get time 0.3610460758 0.8830000000 0.0012640953

1579 3 java get time 0.1510839462 0.6820000000 0.0008111000

1580 3 java get time 0.3630430698 0.8770000000 0.0028901100

1581 3 java get time 0.3541519642 0.8710000000 0.0006880760

1582 3 java get time 0.3691990376 0.9050000000 0.0086929798

1583 3 java get time 0.3516900539 0.8420000000 0.0003039837

1584 3 java get time 0.3529701233 0.9050000000 0.0001938343

1585 3 java get time 0.3504538536 0.8820000000 0.0000500679

1586 3 java get time 0.3573760986 0.8910000000 0.0004990101

1587 3 java get time 0.3511791229 0.8720000000 0.0001299381

1588 3 java get time 0.3628962040 0.9260000000 0.0028851032

1589 3 java get time 0.3614690304 1.0030000000 0.0018129349

1590 3 java get time 0.3563461304 0.9000000000 0.0008018017

1591 3 java get time 0.1569068432 0.7670000000 0.0005040169

1592 3 java get time 0.3652698994 0.9740000000 0.0039172173

1593 3 java get time 0.1553490162 0.6560000000 0.0012388229

1594 3 java get time 0.3563609123 0.8770000000 0.0007479191

1595 3 java get time 0.3566091061 0.8940000000 0.0003819466

1596 3 java get time 0.3577749729 0.8580000000 0.0006880760

1597 3 java get time 0.3511281013 0.8960000000 0.0001912117

1598 3 java get time 0.3523309231 0.9100000000 0.0003230572

1599 3 java get time 0.3571050167 0.8970000000 0.0003440380

1600 3 java get time 0.3540050983 0.8930000000 0.0006680489

1601 3 JavaScript Cookies 0.4008259773 1.0250000000 0.0000109673

1602 3 JavaScript Cookies 0.1486580372 0.4590000000 0.0017940998

1603 3 JavaScript Cookies 0.3507230282 0.8620000000 0.0001230240

1604 3 JavaScript Cookies 0.3537640572 0.8580000000 0.0005550385

1605 3 JavaScript Cookies 0.3611078262 0.6660000000 0.0013558865

1606 3 JavaScript Cookies 0.3583920002 0.9520000000 0.0010170937

1607 3 JavaScript Cookies 0.3586080074 0.8730000000 0.0006098747

1608 3 JavaScript Cookies 0.3544290066 0.8590000000 0.0007860661

1609 3 JavaScript Cookies 0.1526029110 0.7170000000 0.0002181530

1610 3 JavaScript Cookies 0.3501799107 0.8620000000 0.0000221729

1611 3 JavaScript Cookies 0.3435919285 0.6310000000 0.0006959438

1612 3 JavaScript Cookies 0.3585910797 0.9060000000 0.0006291866

1613 3 JavaScript Cookies 0.3537940979 0.9010000000 0.0005531311

1614 3 JavaScript Cookies 0.3448581696 0.8590000000 0.0003659725

1615 3 JavaScript Cookies 0.3627359867 0.9020000000 0.0023028851

1616 3 JavaScript Cookies 0.3586969376 1.0580000000 0.0002322197

1617 3 JavaScript Cookies 0.1510779858 0.6980000000 0.0009100437

1618 3 JavaScript Cookies 0.3646280766 0.8960000000 0.0035688877

1619 3 JavaScript Cookies 0.3578588963 0.9490000000 0.0006639957

1620 3 JavaScript Cookies 0.3488709927 0.6800000000 0.0000858307

1621 3 JavaScript Cookies 0.3521900177 0.8700000000 0.0005598068

1622 3 JavaScript Cookies 0.3528420925 0.8450000000 0.0001373291

1623 3 JavaScript Cookies 0.3452029228 0.8540000000 0.0003342628

1624 3 JavaScript Cookies 0.3513741493 0.8960000000 0.0002510548

Page 100: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXVI

1625 3 JavaScript Cookies 0.3531069756 0.8930000000 0.0001540184

1626 3 JavaScript Cookies 0.3440809250 0.8580000000 0.0007178783

1627 3 JavaScript Cookies 0.3568470478 0.8810000000 0.0002198219

1628 3 JavaScript Cookies 0.1509609222 0.7030000000 0.0003490448

1629 3 JavaScript Cookies 0.3490498066 0.8370000000 0.0000653267

1630 3 JavaScript Cookies 0.3596911430 0.9100000000 0.0001549721

1631 3 JavaScript Cookies 0.3427238464 0.7890000000 0.0014421940

1632 3 JavaScript Cookies 0.3526618481 1.0050000000 0.0001690388

1633 3 JavaScript Cookies 0.3529601097 0.8600000000 0.0002281666

1634 3 JavaScript Cookies 0.3549389839 0.8540000000 0.0004458427

1635 3 JavaScript Cookies 0.3485000134 0.6700000000 0.0000371933

1636 3 JavaScript Cookies 0.3550240993 0.6650000000 0.0003888607

1637 3 JavaScript Cookies 0.3504900932 0.8810000000 0.0000169277

1638 3 JavaScript Cookies 0.3525931835 0.9040000000 0.0000650883

1639 3 JavaScript Cookies 0.3596339226 0.9610000000 0.0000181198

1640 3 JavaScript Cookies 0.1548969746 0.6790000000 0.0008518696

1641 3 JavaScript Cookies 0.3613619804 0.8470000000 0.0015337467

1642 3 JavaScript Cookies 0.3566751480 0.7490000000 0.0003938675

1643 3 JavaScript Cookies 0.3525719643 0.8470000000 0.0000641346

1644 3 JavaScript Cookies 0.3568370342 0.8930000000 0.0002131462

1645 3 JavaScript Cookies 0.3510208130 0.8470000000 0.0001888275

1646 3 JavaScript Cookies 0.1512079239 0.6840000000 0.0008649826

1647 3 JavaScript Cookies 0.1505470276 0.6700000000 0.0005970001

1648 3 JavaScript Cookies 0.1526589394 0.6580000000 0.0001690388

1649 3 JavaScript Cookies 0.1544749737 0.4470000000 0.0002892017

1650 3 JavaScript Cookies 0.3627631664 0.9090000000 0.0022680759

1651 3 JavaScript Cookies 0.3527619839 0.9230000000 0.0001451969

1652 3 JavaScript Cookies 0.3528020382 0.8870000000 0.0001580715

1653 3 JavaScript Cookies 0.3532888889 0.7920000000 0.0002238750

1654 3 JavaScript Cookies 0.3586850166 1.0880000000 0.0005619526

1655 3 JavaScript Cookies 0.3548901081 0.9060000000 0.0004341602

1656 3 JavaScript Cookies 0.3525450230 0.6630000000 0.0001337528

1657 3 JavaScript Cookies 0.3578488827 0.9010000000 0.0006968975

1658 3 JavaScript Cookies 0.3561849594 0.7050000000 0.0007648468

1659 3 JavaScript Cookies 0.3545658588 0.9790000000 0.0007748604

1660 3 JavaScript Cookies 0.3548529148 0.7110000000 0.0003871918

1661 3 JavaScript Cookies 0.3507537842 0.8640000000 0.0000891685

1662 3 JavaScript Cookies 0.1541230679 0.6680000000 0.0001249313

1663 3 JavaScript Cookies 0.3522469997 0.8700000000 0.0002629757

1664 3 JavaScript Cookies 0.3682789803 0.8780000000 0.0061693192

1665 3 JavaScript Cookies 0.3560991287 0.8860000000 0.0008819103

1666 3 JavaScript Cookies 0.3526501656 0.8520000000 0.0000808239

1667 3 JavaScript Cookies 0.3566789627 0.9020000000 0.0003740788

1668 3 JavaScript Cookies 0.3608219624 0.8510000000 0.0005781651

1669 3 JavaScript Cookies 0.3520700932 0.8470000000 0.0005490780

1670 3 JavaScript Cookies 0.3554639816 0.8870000000 0.0005881786

1671 3 JavaScript Cookies 0.1487438679 0.6780000000 0.0017511845

1672 3 JavaScript Cookies 0.3444960117 0.8920000000 0.0003397465

Page 101: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXVII

1673 3 JavaScript Cookies 0.1513130665 0.6880000000 0.0008821487

1674 3 JavaScript Cookies 0.3510458469 0.9570000000 0.0001599789

1675 3 JavaScript Cookies 0.3539779186 0.8930000000 0.0006330013

1676 3 JavaScript Cookies 0.3624749184 0.9310000000 0.0020320415

1677 3 JavaScript Cookies 0.3573780060 0.8810000000 0.0005300045

1678 3 JavaScript Cookies 0.3512110710 0.8490000000 0.0001051426

1679 3 JavaScript Cookies 0.3447670937 0.8730000000 0.0003862381

1680 3 JavaScript Cookies 0.3550739288 0.8760000000 0.0004248619

1681 3 JavaScript Cookies 0.3554239273 0.8640000000 0.0005919933

1682 3 JavaScript Cookies 0.3585431576 0.9350000000 0.0008692741

1683 3 JavaScript Cookies 0.3566989899 0.9250000000 0.0002996922

1684 3 JavaScript Cookies 0.3448438644 0.8600000000 0.0004527569

1685 3 JavaScript Cookies 0.3589959145 0.8740000000 0.0001289845

1686 3 JavaScript Cookies 0.3821680546 0.8940000000 0.0022749901

1687 3 JavaScript Cookies 0.3562440872 1.0020000000 0.0007860661

1688 3 JavaScript Cookies 0.3551628590 0.6800000000 0.0004892349

1689 3 JavaScript Cookies 0.3605818748 0.9120000000 0.0005919933

1690 3 JavaScript Cookies 0.1531291008 0.6780000000 0.0000021458

1691 3 JavaScript Cookies 0.3614449501 0.8860000000 0.0017268658

1692 3 JavaScript Cookies 0.3722589016 0.7050000000 0.0090799332

1693 3 JavaScript Cookies 0.3521041870 0.8540000000 0.0005152225

1694 3 JavaScript Cookies 0.1524689198 0.6830000000 0.0001218319

1695 3 JavaScript Cookies 0.1526010036 0.6510000000 0.0002582073

1696 3 JavaScript Cookies 0.3552281857 0.8740000000 0.0004999638

1697 3 JavaScript Cookies 0.3593711853 0.9010000000 0.0002257824

1698 3 JavaScript Cookies 0.3523540497 0.6570000000 0.0002548695

1699 3 JavaScript Cookies 0.1545059681 0.6990000000 0.0003969669

1700 3 JavaScript Cookies 0.3538980484 0.9590000000 0.0006129742

1701 3 JavaScript Cookies 0.3541419506 0.8570000000 0.0006599426

1702 3 JavaScript Cookies 0.3526279926 0.9270000000 0.0000770092

1703 3 JavaScript Cookies 0.3550369740 0.9800000000 0.0003881454

1704 3 JavaScript Cookies 0.3541328907 0.8790000000 0.0006701946

1705 3 JavaScript Cookies 0.3430781364 0.6770000000 0.0009930134

1706 3 JavaScript Cookies 0.3624949455 0.8800000000 0.0021240711

1707 3 JavaScript Cookies 0.3717510700 0.7020000000 0.0086176395

1708 3 JavaScript Cookies 0.3524470329 0.8730000000 0.0002779961

1709 3 JavaScript Cookies 0.3645520210 0.9130000000 0.0022780895

1710 3 JavaScript Cookies 0.3525700569 0.8540000000 0.0000638962

1711 3 JavaScript Cookies 0.1514389515 0.6370000000 0.0008411407

1712 3 JavaScript Cookies 0.3505389690 0.8440000000 0.0000061989

1713 3 JavaScript Cookies 0.1508829594 0.6790000000 0.0004470348

1714 3 JavaScript Cookies 0.3517110348 0.9300000000 0.0003702641

1715 3 JavaScript Cookies 0.3558490276 0.8740000000 0.0007998943

1716 3 JavaScript Cookies 0.3521261215 0.9100000000 0.0005631447

1717 3 JavaScript Cookies 0.3426108360 0.8750000000 0.0016582012

1718 3 JavaScript Cookies 0.3511869907 0.8530000000 0.0001270771

1719 3 JavaScript Cookies 0.3492407799 0.8620000000 0.0000259876

1720 3 JavaScript Cookies 0.3594307899 0.9970000000 0.0000379086

Page 102: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXVIII

1721 3 JavaScript Cookies 0.3583149910 0.9890000000 0.0009200573

1722 3 JavaScript Cookies 0.3607661724 0.8960000000 0.0005259514

1723 3 JavaScript Cookies 0.1479580402 0.7830000000 0.0023119450

1724 3 JavaScript Cookies 0.1494340897 0.4910000000 0.0008981228

1725 3 JavaScript Cookies 0.3498189449 0.8690000000 0.0002429485

1726 3 JavaScript Cookies 0.3588590622 0.7260000000 0.0000507832

1727 3 JavaScript Cookies 0.1545209885 0.6660000000 0.0004379749

1728 3 JavaScript Cookies 0.3544869423 0.8820000000 0.0007467270

1729 3 JavaScript Cookies 0.3504819870 0.8630000000 0.0000081062

1730 3 JavaScript Cookies 0.1532950401 0.6750000000 0.0002946854

1731 3 JavaScript Cookies 0.3554201126 0.7800000000 0.0005681515

1732 3 JavaScript Cookies 0.3563909531 0.8930000000 0.0006349087

1733 3 JavaScript Cookies 0.3557250500 0.9030000000 0.0007333755

1734 3 JavaScript Cookies 0.3504691124 0.8570000000 0.0000009537

1735 3 JavaScript Cookies 0.1501650810 0.6990000000 0.0006818771

1736 3 JavaScript Cookies 0.3524580002 0.6650000000 0.0002689362

1737 3 JavaScript Cookies 0.3500599861 0.6690000000 0.0000762939

1738 3 JavaScript Cookies 0.3556168079 0.9370000000 0.0006439686

1739 3 JavaScript Cookies 0.3602819443 0.6580000000 0.0002117157

1740 3 JavaScript Cookies 0.1551918983 0.4950000000 0.0009388924

1741 3 JavaScript Cookies 0.1488409042 0.6670000000 0.0013999939

1742 3 JavaScript Cookies 0.3546719551 0.6780000000 0.0006847382

1743 3 JavaScript Cookies 0.3587808609 0.8730000000 0.0000691414

1744 3 JavaScript Cookies 0.1527419090 0.7670000000 0.0001490116

1745 3 JavaScript Cookies 0.3683578968 1.0030000000 0.0081238747

1746 3 JavaScript Cookies 0.3604359627 0.8710000000 0.0002419949

1747 3 JavaScript Cookies 0.3566880226 0.9120000000 0.0003762245

1748 3 JavaScript Cookies 0.3657379150 1.1070000000 0.0041809082

1749 3 JavaScript Cookies 0.3491859436 0.6610000000 0.0000042915

1750 3 JavaScript Cookies 0.1573588848 0.8130000000 0.0005660057

1751 3 JavaScript Cookies 0.1588819027 0.7180000000 0.0010788441

1752 3 JavaScript Cookies 0.1529138088 0.7930000000 0.0000269413

1753 3 JavaScript Cookies 0.3571248055 0.8920000000 0.0003778934

1754 3 JavaScript Cookies 0.3547568321 0.8950000000 0.0004241467

1755 3 JavaScript Cookies 0.3667809963 0.8830000000 0.0050160885

1756 3 JavaScript Cookies 0.3511979580 0.8950000000 0.0001120567

1757 3 JavaScript Cookies 0.3563859463 0.6770000000 0.0006721020

1758 3 JavaScript Cookies 0.1589541435 0.6780000000 0.0011289120

1759 3 JavaScript Cookies 0.3617248535 0.8820000000 0.0018439293

1760 3 JavaScript Cookies 0.3565709591 0.8620000000 0.0004000664

1761 3 JavaScript Cookies 0.3574969769 0.9170000000 0.0005280972

1762 3 JavaScript Cookies 0.3565380573 0.8880000000 0.0004060268

1763 3 JavaScript Cookies 0.3528451920 0.8870000000 0.0001571178

1764 3 JavaScript Cookies 0.3529260159 0.9690000000 0.0002200603

1765 3 JavaScript Cookies 0.3525741100 0.8820000000 0.0000643730

1766 3 JavaScript Cookies 0.3565361500 1.0110000000 0.0004642010

1767 3 JavaScript Cookies 0.3568267822 0.9900000000 0.0002179146

1768 3 JavaScript Cookies 0.3546690941 0.8890000000 0.0006859303

Page 103: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXIX

1769 3 JavaScript Cookies 0.3729529381 0.8910000000 0.0072839260

1770 3 JavaScript Cookies 0.3539290428 0.8830000000 0.0006010532

1771 3 JavaScript Cookies 0.3507330418 0.6990000000 0.0000860691

1772 3 JavaScript Cookies 0.3532860279 0.8710000000 0.0002098083

1773 3 JavaScript Cookies 0.3449580669 0.8520000000 0.0002021790

1774 3 JavaScript Cookies 0.3442637920 0.8860000000 0.0005078316

1775 3 JavaScript Cookies 0.3523969650 0.8720000000 0.0002989769

1776 3 JavaScript Cookies 0.1600310802 0.7140000000 0.0036149025

1777 3 JavaScript Cookies 0.3583328724 0.8780000000 0.0009601116

1778 3 JavaScript Cookies 0.3556530476 1.0460000000 0.0006532669

1779 3 JavaScript Cookies 0.1525189877 0.6800000000 0.0001337528

1780 3 JavaScript Cookies 0.3455369473 0.8610000000 0.0006849766

1781 3 JavaScript Cookies 0.3556761742 0.9010000000 0.0006451607

1782 3 JavaScript Cookies 0.3441979885 0.8560000000 0.0007302761

1783 3 JavaScript Cookies 0.3525118828 0.8930000000 0.0001997948

1784 3 JavaScript Cookies 0.3549170494 0.8750000000 0.0004088879

1785 3 JavaScript Cookies 0.3504190445 0.8640000000 0.0000388622

1786 3 JavaScript Cookies 0.3605370522 0.6850000000 0.0002200603

1787 3 JavaScript Cookies 0.3530430794 0.9250000000 0.0001888275

1788 3 JavaScript Cookies 0.1514840126 0.7300000000 0.0007548332

1789 3 JavaScript Cookies 0.3467819691 0.6510000000 0.0001749992

1790 3 JavaScript Cookies 0.3542528152 1.0100000000 0.0007750988

1791 3 JavaScript Cookies 0.3543479443 0.8740000000 0.0007889271

1792 3 JavaScript Cookies 0.3507680893 0.9960000000 0.0000751019

1793 3 JavaScript Cookies 0.3546879292 0.9290000000 0.0006098747

1794 3 JavaScript Cookies 0.3563640118 0.9130000000 0.0006711483

1795 3 JavaScript Cookies 0.3514409065 0.8570000000 0.0002353191

1796 3 JavaScript Cookies 0.3639509678 0.8840000000 0.0020811558

1797 3 JavaScript Cookies 0.1525790691 0.6930000000 0.0001630783

1798 3 JavaScript Cookies 0.3574168682 0.9140000000 0.0005571842

1799 3 JavaScript Cookies 0.3574850559 0.9040000000 0.0005509853

1800 3 JavaScript Cookies 0.3588979244 0.8920000000 0.0000760555

Page 104: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXX

Appendix N - Regular Solr Measurement Results

Measurement results without the clickthrough data algorithm activated for the efficiency

evaluation.

Index sizes: 1 = 5000 documents, 2 = 50 000 documents, 3 = 1 000 000 documents

Query Number Index Size Query Solr Response Time (s) Site Load Time (s)

1 1 windows install 0.3936569691 0.9280000000

2 1 windows install 0.1903898716 0.7060000000

3 1 windows install 0.4024980068 0.9370000000

4 1 windows install 0.3905069828 0.9160000000

5 1 windows install 0.3926460743 0.9120000000

6 1 windows install 0.3904550076 0.6920000000

7 1 windows install 0.3928749561 0.9110000000

8 1 windows install 0.4188251495 0.9350000000

9 1 windows install 0.4088511467 0.7700000000

10 1 windows install 0.3928229809 0.8750000000

11 1 windows install 0.3906250000 0.6980000000

12 1 windows install 0.1904168129 0.5580000000

13 1 windows install 0.3871150017 1.0600000000

14 1 windows install 0.1908941269 0.7060000000

15 1 windows install 0.3911471367 0.9070000000

16 1 windows install 0.3925960064 0.9260000000

17 1 windows install 0.3974649906 0.9670000000

18 1 windows install 0.3971090317 0.9170000000

19 1 windows install 0.3966839314 0.9300000000

20 1 windows install 0.3968629837 0.9080000000

21 1 windows install 0.3926339149 0.8840000000

22 1 windows install 0.3894419670 0.9180000000

23 1 windows install 0.3868288994 0.9120000000

24 1 windows install 0.3888549805 0.8450000000

25 1 windows install 0.3930890560 0.9590000000

26 1 windows install 0.3948819637 0.9130000000

27 1 windows install 0.3922679424 0.7270000000

28 1 windows install 0.3882880211 0.7350000000

29 1 windows install 0.4111220837 0.9370000000

30 1 windows install 0.3966469765 0.6960000000

31 1 windows install 0.4016468525 0.9480000000

32 1 windows install 0.3871412277 0.9220000000

33 1 windows install 0.3903379440 0.9250000000

34 1 windows install 0.1866858006 0.7490000000

35 1 windows install 0.3968458176 0.8880000000

36 1 windows install 0.3964409828 0.6930000000

37 1 windows install 0.3905808926 1.0600000000

38 1 windows install 0.3990731239 0.9550000000

39 1 windows install 0.3843610287 0.9290000000

40 1 windows install 0.3950750828 0.9310000000

41 1 windows install 0.4072501659 0.9560000000

Page 105: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXI

42 1 windows install 0.3900470734 0.7380000000

43 1 windows install 0.3963539600 0.9090000000

44 1 windows install 0.3937439919 0.8970000000

45 1 windows install 0.1869840622 0.7060000000

46 1 windows install 0.3880729675 0.7080000000

47 1 windows install 0.3932218552 0.7260000000

48 1 windows install 0.3931229115 0.9060000000

49 1 windows install 0.3898699284 0.8840000000

50 1 windows install 0.3984150887 0.9300000000

51 1 windows install 0.3898589611 1.0700000000

52 1 windows install 0.3994309902 0.9490000000

53 1 windows install 0.3780601025 0.8880000000

54 1 windows install 0.3850219250 0.9210000000

55 1 windows install 0.3983998299 0.9160000000

56 1 windows install 0.3967671394 0.9300000000

57 1 windows install 0.3903560638 0.7080000000

58 1 windows install 0.3916358948 0.8870000000

59 1 windows install 0.3901770115 0.9110000000

60 1 windows install 0.1866431236 0.7260000000

61 1 windows install 0.3907630444 0.9370000000

62 1 windows install 0.3916909695 0.9380000000

63 1 windows install 0.3866519928 0.9170000000

64 1 windows install 0.1902899742 0.7080000000

65 1 windows install 0.3905630112 0.9470000000

66 1 windows install 0.3899359703 0.9190000000

67 1 windows install 0.1925520897 0.7550000000

68 1 windows install 0.1907241344 0.7150000000

69 1 windows install 0.3886539936 0.9920000000

70 1 windows install 0.3920519352 0.9070000000

71 1 windows install 0.3965120316 0.9010000000

72 1 windows install 0.3907010555 0.9150000000

73 1 windows install 0.3880360126 0.9280000000

74 1 windows install 0.3937518597 0.9400000000

75 1 windows install 0.3964140415 0.9280000000

76 1 windows install 0.4006819725 0.8960000000

77 1 windows install 0.3875110149 0.9510000000

78 1 windows install 0.3942530155 0.9560000000

79 1 windows install 0.4010000229 0.9160000000

80 1 windows install 0.3995268345 0.8920000000

81 1 windows install 0.3932709694 0.7060000000

82 1 windows install 0.3927860260 0.9110000000

83 1 windows install 0.3890950680 0.9100000000

84 1 windows install 0.4008269310 0.9310000000

85 1 windows install 0.1890521049 0.7690000000

86 1 windows install 0.3907060623 0.9170000000

87 1 windows install 0.4006252289 0.9530000000

88 1 windows install 0.3976590633 0.7990000000

89 1 windows install 0.3864219189 0.7190000000

Page 106: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXII

90 1 windows install 0.4015989304 0.9140000000

91 1 windows install 0.3945748806 0.9300000000

92 1 windows install 0.3955509663 0.9210000000

93 1 windows install 0.1942639351 0.6790000000

94 1 windows install 0.3965299129 0.9180000000

95 1 windows install 0.3947379589 1.0180000000

96 1 windows install 0.4014852047 0.9260000000

97 1 windows install 0.3909530640 1.0490000000

98 1 windows install 0.3971939087 0.9250000000

99 1 windows install 0.4029390812 0.9050000000

100 1 windows install 0.3900752068 0.8840000000

101 1 windows install 0.3903532028 0.6940000000

102 1 windows install 0.3885710239 1.0490000000

103 1 windows install 0.2022318840 0.7160000000

104 1 windows install 0.4004931450 0.9220000000

105 1 windows install 0.3949759007 0.7490000000

106 1 windows install 0.1907911301 0.6990000000

107 1 windows install 0.4050498009 0.9670000000

108 1 windows install 0.3921689987 0.7190000000

109 1 windows install 0.3840930462 0.6710000000

110 1 windows install 0.4028232098 0.9280000000

111 1 windows install 0.3919279575 0.9430000000

112 1 windows install 0.3981759548 0.9120000000

113 1 windows install 0.3986392021 0.9490000000

114 1 windows install 0.1948080063 0.6990000000

115 1 windows install 0.3971271515 0.9140000000

116 1 windows install 0.3984229565 0.7090000000

117 1 windows install 0.3906459808 0.9350000000

118 1 windows install 0.4065079689 1.0740000000

119 1 windows install 0.3903238773 0.9180000000

120 1 windows install 0.3972299099 0.9060000000

121 1 windows install 0.1990160942 0.6940000000

122 1 windows install 0.4005849361 0.9200000000

123 1 windows install 0.3946528435 0.9370000000

124 1 windows install 0.3901059628 0.9100000000

125 1 windows install 0.3951430321 0.9340000000

126 1 windows install 0.3950228691 0.9400000000

127 1 windows install 0.3848161697 0.9090000000

128 1 windows install 0.3969159126 0.7030000000

129 1 windows install 0.3955140114 0.7300000000

130 1 windows install 0.3945472240 0.9120000000

131 1 windows install 0.3946111202 0.9210000000

132 1 windows install 0.3883550167 0.9030000000

133 1 windows install 0.3932750225 0.9700000000

134 1 windows install 0.3843441010 0.8980000000

135 1 windows install 0.3932230473 0.9080000000

136 1 windows install 0.3892209530 0.9500000000

137 1 windows install 0.3907589912 0.9170000000

Page 107: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXIII

138 1 windows install 0.1925821304 0.7350000000

139 1 windows install 0.1974430084 0.7340000000

140 1 windows install 0.3932399750 0.8940000000

141 1 windows install 0.1944439411 0.7300000000

142 1 windows install 0.3947918415 0.9210000000

143 1 windows install 0.3989820480 0.9600000000

144 1 windows install 0.3996200562 0.9320000000

145 1 windows install 0.1902930737 0.7050000000

146 1 windows install 0.3923881054 0.9050000000

147 1 windows install 0.3886640072 0.7230000000

148 1 windows install 0.3843300343 0.9330000000

149 1 windows install 0.3905808926 0.9100000000

150 1 windows install 0.3925549984 0.9170000000

151 1 windows install 0.3926839828 0.9000000000

152 1 windows install 0.3929939270 0.9290000000

153 1 windows install 0.3846759796 0.8780000000

154 1 windows install 0.3977429867 0.9130000000

155 1 windows install 0.4027810097 0.9240000000

156 1 windows install 0.3926460743 0.7240000000

157 1 windows install 0.3944530487 0.9120000000

158 1 windows install 0.3890149593 0.7040000000

159 1 windows install 0.3911001682 0.9070000000

160 1 windows install 0.1906011105 0.5410000000

161 1 windows install 0.1930649281 0.7600000000

162 1 windows install 0.3892540932 0.9070000000

163 1 windows install 0.3942039013 0.9090000000

164 1 windows install 0.3946921825 0.9160000000

165 1 windows install 0.3926959038 0.9570000000

166 1 windows install 0.3908338547 0.9210000000

167 1 windows install 0.3949868679 0.9060000000

168 1 windows install 0.4230961800 0.9630000000

169 1 windows install 0.3998420238 0.9670000000

170 1 windows install 0.3935549259 0.9160000000

171 1 windows install 0.3968420029 0.9150000000

172 1 windows install 0.3947529793 0.9140000000

173 1 windows install 0.3909041882 0.9080000000

174 1 windows install 0.3869159222 0.9280000000

175 1 windows install 0.3944270611 0.9060000000

176 1 windows install 0.3948640823 0.9090000000

177 1 windows install 0.3907339573 0.9180000000

178 1 windows install 0.3891041279 0.9100000000

179 1 windows install 0.1947598457 0.6940000000

180 1 windows install 0.3986890316 0.9110000000

181 1 windows install 0.3889379501 0.8920000000

182 1 windows install 0.3923721313 0.9430000000

183 1 windows install 0.3916649818 0.8940000000

184 1 windows install 0.3927588463 0.9850000000

185 1 windows install 0.3957660198 0.9310000000

Page 108: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXIV

186 1 windows install 0.3928380013 0.9410000000

187 1 windows install 0.3905460835 0.9160000000

188 1 windows install 0.3946971893 0.9260000000

189 1 windows install 0.3901181221 0.9200000000

190 1 windows install 0.3962550163 0.9260000000

191 1 windows install 0.4070250988 0.9040000000

192 1 windows install 0.3907449245 0.9200000000

193 1 windows install 0.3997030258 0.9080000000

194 1 windows install 0.1868929863 0.7150000000

195 1 windows install 0.1846728325 0.7180000000

196 1 windows install 0.1902019978 0.6950000000

197 1 windows install 0.3886749744 0.7360000000

198 1 windows install 0.3950459957 0.9400000000

199 1 windows install 0.1908249855 0.5090000000

200 1 windows install 0.3885591030 0.7100000000

201 1 java get time 0.3835740089 0.8780000000

202 1 java get time 0.3884649277 0.8910000000

203 1 java get time 0.3869609833 0.6940000000

204 1 java get time 0.3868529797 0.9530000000

205 1 java get time 0.3958749771 0.9240000000

206 1 java get time 0.3986279964 0.9180000000

207 1 java get time 0.1862471104 0.6910000000

208 1 java get time 0.3785789013 0.9240000000

209 1 java get time 0.3946700096 0.9080000000

210 1 java get time 0.3950159550 0.9400000000

211 1 java get time 0.1884579659 0.7310000000

212 1 java get time 0.3858721256 0.8960000000

213 1 java get time 0.3984470367 0.6900000000

214 1 java get time 0.3916618824 0.8810000000

215 1 java get time 0.1886119843 0.7230000000

216 1 java get time 0.3923099041 0.9330000000

217 1 java get time 0.3914580345 0.6800000000

218 1 java get time 0.1908278465 0.6970000000

219 1 java get time 0.4004788399 0.9040000000

220 1 java get time 0.3955399990 0.9190000000

221 1 java get time 0.3936660290 0.9010000000

222 1 java get time 0.3986561298 0.9330000000

223 1 java get time 0.3945810795 0.9450000000

224 1 java get time 0.3828701973 0.8900000000

225 1 java get time 0.3990349770 0.9080000000

226 1 java get time 0.3905868530 0.9170000000

227 1 java get time 0.1886761189 0.6900000000

228 1 java get time 0.4009549618 0.7160000000

229 1 java get time 0.3927531242 0.9500000000

230 1 java get time 0.1885828972 0.5230000000

231 1 java get time 0.3931200504 0.9240000000

232 1 java get time 0.4043228626 0.9440000000

233 1 java get time 0.1926820278 0.7150000000

Page 109: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXV

234 1 java get time 0.4006109238 0.9130000000

235 1 java get time 0.1916151047 0.7210000000

236 1 java get time 0.4005188942 0.9190000000

237 1 java get time 0.3947818279 0.9380000000

238 1 java get time 0.3925862312 0.6990000000

239 1 java get time 0.3942108154 0.7130000000

240 1 java get time 0.3906049728 0.9140000000

241 1 java get time 0.4008529186 0.9290000000

242 1 java get time 0.3904829025 0.9060000000

243 1 java get time 0.3891448975 0.8980000000

244 1 java get time 0.3997409344 0.9310000000

245 1 java get time 0.3941199780 0.9150000000

246 1 java get time 0.3905010223 0.9350000000

247 1 java get time 0.1929290295 0.6980000000

248 1 java get time 0.3946769238 0.7160000000

249 1 java get time 0.3913049698 0.8880000000

250 1 java get time 0.3933451176 0.9050000000

251 1 java get time 0.3884990215 0.9080000000

252 1 java get time 0.4085178375 0.9060000000

253 1 java get time 0.3925218582 0.9100000000

254 1 java get time 0.1885938644 0.5040000000

255 1 java get time 0.3908329010 0.9040000000

256 1 java get time 0.3881630898 0.9470000000

257 1 java get time 0.3922810555 0.9170000000

258 1 java get time 0.3967621326 0.9240000000

259 1 java get time 0.1964068413 0.7090000000

260 1 java get time 0.1905281544 0.7180000000

261 1 java get time 0.3911550045 0.8910000000

262 1 java get time 0.1865251064 0.6970000000

263 1 java get time 0.3966629505 0.9200000000

264 1 java get time 0.3951969147 0.9080000000

265 1 java get time 0.3940730095 0.7390000000

266 1 java get time 0.3987450600 0.9540000000

267 1 java get time 0.3909540176 0.8820000000

268 1 java get time 0.3925640583 0.9530000000

269 1 java get time 0.3886179924 0.9070000000

270 1 java get time 0.4039959908 0.9150000000

271 1 java get time 0.3981008530 0.9050000000

272 1 java get time 0.3969399929 0.9130000000

273 1 java get time 0.3909561634 0.9050000000

274 1 java get time 0.3928620815 0.9210000000

275 1 java get time 0.1863551140 0.6840000000

276 1 java get time 0.3888809681 0.9370000000

277 1 java get time 0.3948349953 0.9080000000

278 1 java get time 0.3942878246 0.9250000000

279 1 java get time 0.3908081055 0.9460000000

280 1 java get time 0.3853318691 0.8860000000

281 1 java get time 0.3906428814 0.9330000000

Page 110: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXVI

282 1 java get time 0.3989958763 0.9110000000

283 1 java get time 0.4103479385 1.0750000000

284 1 java get time 0.1928210258 0.7010000000

285 1 java get time 0.4084579945 0.9250000000

286 1 java get time 0.2040288448 0.7300000000

287 1 java get time 0.1927509308 0.7200000000

288 1 java get time 0.3949470520 0.8940000000

289 1 java get time 0.3951289654 0.9210000000

290 1 java get time 0.3818140030 0.9050000000

291 1 java get time 0.3957381248 0.9130000000

292 1 java get time 0.1907641888 0.7110000000

293 1 java get time 0.3888309002 0.7390000000

294 1 java get time 0.3912808895 0.9270000000

295 1 java get time 0.3927671909 0.8990000000

296 1 java get time 0.4025278091 0.9400000000

297 1 java get time 0.3967449665 0.9190000000

298 1 java get time 0.3880829811 0.9180000000

299 1 java get time 0.3927450180 0.9090000000

300 1 java get time 0.3799920082 1.0550000000

301 1 java get time 0.3917880058 0.9230000000

302 1 java get time 0.3948149681 0.9150000000

303 1 java get time 0.1987838745 0.7010000000

304 1 java get time 0.3925030231 0.9520000000

305 1 java get time 0.3949999809 0.9750000000

306 1 java get time 0.3886020184 0.9540000000

307 1 java get time 0.3909368515 0.9060000000

308 1 java get time 0.3919229507 0.9220000000

309 1 java get time 0.3967120647 0.9260000000

310 1 java get time 0.3925511837 0.9290000000

311 1 java get time 0.4000139236 0.9100000000

312 1 java get time 0.3929088116 0.9130000000

313 1 java get time 0.3885970116 0.8770000000

314 1 java get time 0.3914890289 0.9310000000

315 1 java get time 0.4044620991 1.0590000000

316 1 java get time 0.1879830360 0.7320000000

317 1 java get time 0.3925678730 0.9910000000

318 1 java get time 0.1905381680 0.6780000000

319 1 java get time 0.3959310055 0.9220000000

320 1 java get time 0.3943929672 0.6980000000

321 1 java get time 0.4040839672 0.9000000000

322 1 java get time 0.3966619968 0.8920000000

323 1 java get time 0.3889172077 1.0950000000

324 1 java get time 0.3886559010 0.8990000000

325 1 java get time 0.3867900372 0.8780000000

326 1 java get time 0.3986220360 0.9040000000

327 1 java get time 0.4046909809 0.9320000000

328 1 java get time 0.3974571228 0.9030000000

329 1 java get time 0.4028358459 0.9070000000

Page 111: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXVII

330 1 java get time 0.3907580376 0.9640000000

331 1 java get time 0.4012300968 0.9680000000

332 1 java get time 0.3910028934 1.0270000000

333 1 java get time 0.3967139721 0.7400000000

334 1 java get time 0.3884711266 0.9760000000

335 1 java get time 0.3906030655 0.9290000000

336 1 java get time 0.1904029846 0.5080000000

337 1 java get time 0.3967428207 0.9400000000

338 1 java get time 0.1900901794 0.7040000000

339 1 java get time 0.3927328587 0.9680000000

340 1 java get time 0.3905658722 1.0180000000

341 1 java get time 0.1987400055 0.8020000000

342 1 java get time 0.3971121311 1.0000000000

343 1 java get time 0.3909220695 0.9320000000

344 1 java get time 0.3968949318 1.0140000000

345 1 java get time 0.3926329613 0.9110000000

346 1 java get time 0.1867449284 0.5000000000

347 1 java get time 0.4031488895 0.9040000000

348 1 java get time 0.3975059986 0.9050000000

349 1 java get time 0.4103431702 1.0670000000

350 1 java get time 0.3904929161 0.9070000000

351 1 java get time 0.3911790848 0.9390000000

352 1 java get time 0.3914899826 1.0460000000

353 1 java get time 0.3913359642 0.9070000000

354 1 java get time 0.3925809860 0.9840000000

355 1 java get time 0.4081308842 0.9420000000

356 1 java get time 0.3929281235 0.9100000000

357 1 java get time 0.3902199268 0.8660000000

358 1 java get time 0.1908690929 0.8490000000

359 1 java get time 0.4148068428 1.0560000000

360 1 java get time 0.3905479908 0.9070000000

361 1 java get time 0.3915500641 0.9090000000

362 1 java get time 0.3962728977 0.9470000000

363 1 java get time 0.3908610344 0.9240000000

364 1 java get time 0.3899090290 0.8960000000

365 1 java get time 0.3964970112 0.9270000000

366 1 java get time 0.3937261105 0.9000000000

367 1 java get time 0.3836488724 0.9250000000

368 1 java get time 0.3867619038 1.0240000000

369 1 java get time 0.3965520859 0.9120000000

370 1 java get time 0.3862769604 1.1010000000

371 1 java get time 0.1865739822 0.7270000000

372 1 java get time 0.3886549473 0.9140000000

373 1 java get time 0.4006369114 0.6940000000

374 1 java get time 0.3912911415 0.9120000000

375 1 java get time 0.3944549561 0.9510000000

376 1 java get time 0.3863079548 0.9580000000

377 1 java get time 0.3986639977 0.7100000000

Page 112: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXVIII

378 1 java get time 0.3968431950 0.8840000000

379 1 java get time 0.3990271091 0.8330000000

380 1 java get time 0.3908181190 0.8940000000

381 1 java get time 0.3931989670 1.0120000000

382 1 java get time 0.3924081326 0.9740000000

383 1 java get time 0.4085829258 1.0120000000

384 1 java get time 0.3946669102 0.9410000000

385 1 java get time 0.3925549984 0.9280000000

386 1 java get time 0.3888061047 0.8930000000

387 1 java get time 0.3926639557 0.9060000000

388 1 java get time 0.3910038471 0.8090000000

389 1 java get time 0.4008409977 0.9370000000

390 1 java get time 0.3947639465 0.9150000000

391 1 java get time 0.4109790325 1.0390000000

392 1 java get time 0.4089989662 0.9230000000

393 1 java get time 0.4082598686 0.9500000000

394 1 java get time 0.3867919445 0.8920000000

395 1 java get time 0.3935329914 0.9750000000

396 1 java get time 0.3956348896 0.8800000000

397 1 java get time 0.3905808926 0.9100000000

398 1 java get time 0.3925518990 0.9120000000

399 1 java get time 0.4008500576 0.9110000000

400 1 java get time 0.4062337875 0.9310000000

401 1 JavaScript Cookies 0.3182671070 0.6170000000

402 1 JavaScript Cookies 0.3160769939 0.8160000000

403 1 JavaScript Cookies 0.3151848316 0.8100000000

404 1 JavaScript Cookies 0.3163089752 0.8590000000

405 1 JavaScript Cookies 0.3163259029 0.9620000000

406 1 JavaScript Cookies 0.3161818981 0.6700000000

407 1 JavaScript Cookies 0.3135960102 0.8360000000

408 1 JavaScript Cookies 0.1119890213 0.4390000000

409 1 JavaScript Cookies 0.3157649040 0.8410000000

410 1 JavaScript Cookies 0.3242990971 0.8210000000

411 1 JavaScript Cookies 0.3161170483 0.8230000000

412 1 JavaScript Cookies 0.3151528835 0.8380000000

413 1 JavaScript Cookies 0.3160879612 0.8230000000

414 1 JavaScript Cookies 0.3162441254 0.8250000000

415 1 JavaScript Cookies 0.3180878162 0.8390000000

416 1 JavaScript Cookies 0.3155541420 0.8070000000

417 1 JavaScript Cookies 0.3081328869 0.6160000000

418 1 JavaScript Cookies 0.3318481445 0.8510000000

419 1 JavaScript Cookies 0.3138489723 0.8230000000

420 1 JavaScript Cookies 0.3158531189 0.8290000000

421 1 JavaScript Cookies 0.3169851303 0.6350000000

422 1 JavaScript Cookies 0.1141021252 0.7350000000

423 1 JavaScript Cookies 0.3165481091 0.6240000000

424 1 JavaScript Cookies 0.1143860817 0.6100000000

425 1 JavaScript Cookies 0.3169720173 0.8340000000

Page 113: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXIX

426 1 JavaScript Cookies 0.1118018627 0.6380000000

427 1 JavaScript Cookies 0.3143508434 0.8070000000

428 1 JavaScript Cookies 0.1134061813 0.6440000000

429 1 JavaScript Cookies 0.3164398670 0.8260000000

430 1 JavaScript Cookies 0.1163489819 0.6910000000

431 1 JavaScript Cookies 0.3245329857 0.8680000000

432 1 JavaScript Cookies 0.3177559376 0.8520000000

433 1 JavaScript Cookies 0.1138639450 0.7090000000

434 1 JavaScript Cookies 0.3197240829 0.8280000000

435 1 JavaScript Cookies 0.3184530735 0.8180000000

436 1 JavaScript Cookies 0.1158759594 0.6140000000

437 1 JavaScript Cookies 0.3161118031 0.8410000000

438 1 JavaScript Cookies 0.3222568035 0.8310000000

439 1 JavaScript Cookies 0.3178579807 0.8260000000

440 1 JavaScript Cookies 0.3137700558 0.8420000000

441 1 JavaScript Cookies 0.3149909973 0.8420000000

442 1 JavaScript Cookies 0.1100230217 0.5960000000

443 1 JavaScript Cookies 0.3161458969 0.8290000000

444 1 JavaScript Cookies 0.3123600483 0.9420000000

445 1 JavaScript Cookies 0.3199310303 0.8320000000

446 1 JavaScript Cookies 0.3145620823 0.6370000000

447 1 JavaScript Cookies 0.3062930107 0.9490000000

448 1 JavaScript Cookies 0.3178510666 0.8620000000

449 1 JavaScript Cookies 0.3140099049 0.8410000000

450 1 JavaScript Cookies 0.3229649067 0.6830000000

451 1 JavaScript Cookies 0.3125879765 0.8230000000

452 1 JavaScript Cookies 0.3147602081 0.8420000000

453 1 JavaScript Cookies 0.3229150772 0.6130000000

454 1 JavaScript Cookies 0.3156011105 0.6310000000

455 1 JavaScript Cookies 0.3141300678 0.8760000000

456 1 JavaScript Cookies 0.3144340515 0.9120000000

457 1 JavaScript Cookies 0.3216638565 0.8340000000

458 1 JavaScript Cookies 0.3160719872 0.8440000000

459 1 JavaScript Cookies 0.3143579960 0.8140000000

460 1 JavaScript Cookies 0.1126308441 0.6270000000

461 1 JavaScript Cookies 0.3125610352 0.5970000000

462 1 JavaScript Cookies 0.3187510967 0.9180000000

463 1 JavaScript Cookies 0.3147759438 0.8490000000

464 1 JavaScript Cookies 0.3202660084 0.8370000000

465 1 JavaScript Cookies 0.3164060116 0.8200000000

466 1 JavaScript Cookies 0.3121030331 0.9350000000

467 1 JavaScript Cookies 0.3179471493 0.8320000000

468 1 JavaScript Cookies 0.3140590191 0.8140000000

469 1 JavaScript Cookies 0.3184280396 0.8270000000

470 1 JavaScript Cookies 0.3115901947 0.8890000000

471 1 JavaScript Cookies 0.3196899891 0.8370000000

472 1 JavaScript Cookies 0.3155090809 0.8410000000

473 1 JavaScript Cookies 0.3174159527 0.8110000000

Page 114: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXX

474 1 JavaScript Cookies 0.3181240559 0.8470000000

475 1 JavaScript Cookies 0.3059909344 0.8410000000

476 1 JavaScript Cookies 0.1134190559 0.6130000000

477 1 JavaScript Cookies 0.1108961105 0.6130000000

478 1 JavaScript Cookies 0.3120620251 0.8110000000

479 1 JavaScript Cookies 0.3142318726 0.9000000000

480 1 JavaScript Cookies 0.3197281361 0.8320000000

481 1 JavaScript Cookies 0.3144929409 0.8400000000

482 1 JavaScript Cookies 0.1121690273 0.6290000000

483 1 JavaScript Cookies 0.3297550678 0.8290000000

484 1 JavaScript Cookies 0.3103399277 0.8880000000

485 1 JavaScript Cookies 0.3389208317 0.8090000000

486 1 JavaScript Cookies 0.3100879192 0.8380000000

487 1 JavaScript Cookies 0.3118939400 0.8880000000

488 1 JavaScript Cookies 0.1100981236 0.6520000000

489 1 JavaScript Cookies 0.3162939548 0.7970000000

490 1 JavaScript Cookies 0.3145499229 0.8130000000

491 1 JavaScript Cookies 0.1140830517 0.6240000000

492 1 JavaScript Cookies 0.3120510578 0.8910000000

493 1 JavaScript Cookies 0.3362209797 0.9500000000

494 1 JavaScript Cookies 0.3176169395 0.8340000000

495 1 JavaScript Cookies 0.3156640530 0.6310000000

496 1 JavaScript Cookies 0.3148031235 0.7950000000

497 1 JavaScript Cookies 0.3266341686 0.8430000000

498 1 JavaScript Cookies 0.3138730526 0.8290000000

499 1 JavaScript Cookies 0.3189780712 0.6030000000

500 1 JavaScript Cookies 0.3123419285 0.7970000000

501 1 JavaScript Cookies 0.3154079914 0.8230000000

502 1 JavaScript Cookies 0.1117839813 0.5870000000

503 1 JavaScript Cookies 0.3221778870 0.9090000000

504 1 JavaScript Cookies 0.3171129227 0.9300000000

505 1 JavaScript Cookies 0.3184328079 0.8210000000

506 1 JavaScript Cookies 0.3197159767 0.8340000000

507 1 JavaScript Cookies 0.3161621094 0.8420000000

508 1 JavaScript Cookies 0.3080019951 0.8950000000

509 1 JavaScript Cookies 0.3244719505 0.8330000000

510 1 JavaScript Cookies 0.3083889484 0.8220000000

511 1 JavaScript Cookies 0.3060569763 0.8240000000

512 1 JavaScript Cookies 0.1188220978 0.6200000000

513 1 JavaScript Cookies 0.3137750626 0.8100000000

514 1 JavaScript Cookies 0.3139870167 0.9100000000

515 1 JavaScript Cookies 0.1143610477 0.6420000000

516 1 JavaScript Cookies 0.3147079945 0.9040000000

517 1 JavaScript Cookies 0.3161671162 0.6350000000

518 1 JavaScript Cookies 0.3178730011 0.9070000000

519 1 JavaScript Cookies 0.3168320656 0.8480000000

520 1 JavaScript Cookies 0.3180139065 0.8510000000

521 1 JavaScript Cookies 0.3157689571 0.8120000000

Page 115: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXI

522 1 JavaScript Cookies 0.3150939941 0.8400000000

523 1 JavaScript Cookies 0.3077230453 0.8240000000

524 1 JavaScript Cookies 0.3079881668 0.8350000000

525 1 JavaScript Cookies 0.3162670135 0.8630000000

526 1 JavaScript Cookies 0.3117599487 0.8430000000

527 1 JavaScript Cookies 0.3260629177 0.6190000000

528 1 JavaScript Cookies 0.3163402081 0.8390000000

529 1 JavaScript Cookies 0.3176410198 0.8170000000

530 1 JavaScript Cookies 0.3186569214 0.8340000000

531 1 JavaScript Cookies 0.3206429482 0.9050000000

532 1 JavaScript Cookies 0.3165299892 0.8650000000

533 1 JavaScript Cookies 0.3121531010 0.8200000000

534 1 JavaScript Cookies 0.3145060539 0.8330000000

535 1 JavaScript Cookies 0.3240408897 0.8470000000

536 1 JavaScript Cookies 0.3259711266 0.9410000000

537 1 JavaScript Cookies 0.3059499264 0.8380000000

538 1 JavaScript Cookies 0.3184769154 0.6240000000

539 1 JavaScript Cookies 0.3196110725 0.8310000000

540 1 JavaScript Cookies 0.3143270016 0.8090000000

541 1 JavaScript Cookies 0.3281168938 0.9370000000

542 1 JavaScript Cookies 0.3168358803 0.8130000000

543 1 JavaScript Cookies 0.3159770966 0.8200000000

544 1 JavaScript Cookies 0.3100240231 0.8220000000

545 1 JavaScript Cookies 0.1139209270 0.6370000000

546 1 JavaScript Cookies 0.3157520294 0.8150000000

547 1 JavaScript Cookies 0.3225471973 0.8410000000

548 1 JavaScript Cookies 0.3163449764 0.8050000000

549 1 JavaScript Cookies 0.3181490898 0.8390000000

550 1 JavaScript Cookies 0.3044130802 0.8010000000

551 1 JavaScript Cookies 0.3199570179 0.8700000000

552 1 JavaScript Cookies 0.3353800774 0.8580000000

553 1 JavaScript Cookies 0.3118779659 0.8160000000

554 1 JavaScript Cookies 0.1160678864 0.6230000000

555 1 JavaScript Cookies 0.3242838383 0.8270000000

556 1 JavaScript Cookies 0.3138899803 0.9510000000

557 1 JavaScript Cookies 0.3178989887 0.8440000000

558 1 JavaScript Cookies 0.3199589252 0.8180000000

559 1 JavaScript Cookies 0.3161261082 0.6200000000

560 1 JavaScript Cookies 0.3076219559 0.8310000000

561 1 JavaScript Cookies 0.3168818951 0.8330000000

562 1 JavaScript Cookies 0.3141098022 0.8150000000

563 1 JavaScript Cookies 0.3261761665 0.8260000000

564 1 JavaScript Cookies 0.3159821033 0.8100000000

565 1 JavaScript Cookies 0.3235268593 0.6430000000

566 1 JavaScript Cookies 0.1117238998 0.6570000000

567 1 JavaScript Cookies 0.3118679523 0.6340000000

568 1 JavaScript Cookies 0.3199241161 0.8390000000

569 1 JavaScript Cookies 0.3152391911 0.8230000000

Page 116: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXII

570 1 JavaScript Cookies 0.3120348454 0.8050000000

571 1 JavaScript Cookies 0.3118770123 0.8870000000

572 1 JavaScript Cookies 0.3163068295 0.8270000000

573 1 JavaScript Cookies 0.3199090958 0.8240000000

574 1 JavaScript Cookies 0.1201038361 0.6530000000

575 1 JavaScript Cookies 0.3035719395 0.7970000000

576 1 JavaScript Cookies 0.3138120174 0.8240000000

577 1 JavaScript Cookies 0.3138818741 0.8380000000

578 1 JavaScript Cookies 0.3166670799 0.9660000000

579 1 JavaScript Cookies 0.3123610020 0.8260000000

580 1 JavaScript Cookies 0.3106589317 0.8250000000

581 1 JavaScript Cookies 0.3206400871 0.8350000000

582 1 JavaScript Cookies 0.1142070293 0.6340000000

583 1 JavaScript Cookies 0.3179919720 0.8460000000

584 1 JavaScript Cookies 0.3060209751 0.8440000000

585 1 JavaScript Cookies 0.3050479889 0.6090000000

586 1 JavaScript Cookies 0.3320288658 0.8390000000

587 1 JavaScript Cookies 0.3157501221 0.8170000000

588 1 JavaScript Cookies 0.3335251808 0.8320000000

589 1 JavaScript Cookies 0.3121640682 0.8170000000

590 1 JavaScript Cookies 0.3140451908 0.8040000000

591 1 JavaScript Cookies 0.1136209965 0.7160000000

592 1 JavaScript Cookies 0.3138718605 0.5970000000

593 1 JavaScript Cookies 0.3157758713 0.8480000000

594 1 JavaScript Cookies 0.3163189888 0.8130000000

595 1 JavaScript Cookies 0.3180348873 0.8490000000

596 1 JavaScript Cookies 0.3186841011 0.8170000000

597 1 JavaScript Cookies 0.3167850971 0.8410000000

598 1 JavaScript Cookies 0.1155929565 0.6300000000

599 1 JavaScript Cookies 0.1141250134 0.6270000000

600 1 JavaScript Cookies 0.3274900913 0.8420000000

601 2 windows install 0.3874158859 0.7140000000

602 2 windows install 0.1884360313 0.7400000000

603 2 windows install 0.1908321381 0.7470000000

604 2 windows install 0.3972160816 0.9240000000

605 2 windows install 0.1969351768 0.7120000000

606 2 windows install 0.3883769512 0.9440000000

607 2 windows install 0.3967680931 0.9510000000

608 2 windows install 0.1902811527 0.7460000000

609 2 windows install 0.3944740295 0.9120000000

610 2 windows install 0.3922140598 0.9680000000

611 2 windows install 0.3928909302 0.9650000000

612 2 windows install 0.3982880116 0.9270000000

613 2 windows install 0.3824319839 0.9270000000

614 2 windows install 0.4010169506 0.9300000000

615 2 windows install 0.3868300915 0.7440000000

616 2 windows install 0.3957791328 0.9430000000

617 2 windows install 0.3882091045 0.6980000000

Page 117: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXIII

618 2 windows install 0.3965919018 0.8950000000

619 2 windows install 0.3927559853 0.9300000000

620 2 windows install 0.3948059082 0.9530000000

621 2 windows install 0.3926401138 0.9210000000

622 2 windows install 0.3929750919 0.9030000000

623 2 windows install 0.3928351402 0.9320000000

624 2 windows install 0.4024670124 0.9280000000

625 2 windows install 0.3967142105 0.9620000000

626 2 windows install 0.3951117992 0.9090000000

627 2 windows install 0.4088778496 0.9340000000

628 2 windows install 0.1890289783 0.7060000000

629 2 windows install 0.3879399300 0.9150000000

630 2 windows install 0.3990609646 0.9420000000

631 2 windows install 0.1899900436 0.7070000000

632 2 windows install 0.3888008595 0.9300000000

633 2 windows install 0.3927059174 0.9470000000

634 2 windows install 0.4007151127 0.8850000000

635 2 windows install 0.3970429897 0.8940000000

636 2 windows install 0.3897500038 0.9230000000

637 2 windows install 0.3945488930 0.7300000000

638 2 windows install 0.4005920887 0.9520000000

639 2 windows install 0.3929159641 0.8940000000

640 2 windows install 0.3919620514 0.9100000000

641 2 windows install 0.1902549267 0.5530000000

642 2 windows install 0.3915371895 0.9300000000

643 2 windows install 0.3933820724 0.9410000000

644 2 windows install 0.3948929310 0.6900000000

645 2 windows install 0.1961579323 0.7060000000

646 2 windows install 0.1947588921 0.7950000000

647 2 windows install 0.3923299313 0.9130000000

648 2 windows install 0.3885819912 0.6690000000

649 2 windows install 0.3907051086 0.8990000000

650 2 windows install 0.3861260414 0.9070000000

651 2 windows install 0.4010200500 0.9590000000

652 2 windows install 0.3948099613 0.9260000000

653 2 windows install 0.1910090446 0.7110000000

654 2 windows install 0.3894479275 0.9020000000

655 2 windows install 0.3991179466 0.9180000000

656 2 windows install 0.3923950195 0.7480000000

657 2 windows install 0.3929109573 0.7220000000

658 2 windows install 0.3930931091 0.7030000000

659 2 windows install 0.3905150890 0.9280000000

660 2 windows install 0.3947420120 0.9230000000

661 2 windows install 0.3902959824 0.7360000000

662 2 windows install 0.3908421993 0.9160000000

663 2 windows install 0.3905320168 0.9230000000

664 2 windows install 0.3878569603 0.9140000000

665 2 windows install 0.3923420906 0.9250000000

Page 118: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXIV

666 2 windows install 0.3988800049 0.9950000000

667 2 windows install 0.3886520863 0.7410000000

668 2 windows install 0.3886101246 0.8920000000

669 2 windows install 0.3928749561 0.9220000000

670 2 windows install 0.3939850330 0.9490000000

671 2 windows install 0.3907790184 0.9250000000

672 2 windows install 0.3973100185 0.9440000000

673 2 windows install 0.3968360424 0.9550000000

674 2 windows install 0.4003069401 0.7290000000

675 2 windows install 0.3840110302 0.9180000000

676 2 windows install 0.3929522038 0.9410000000

677 2 windows install 0.3906669617 0.9390000000

678 2 windows install 0.3970408440 0.9060000000

679 2 windows install 0.3929898739 0.9350000000

680 2 windows install 0.3931319714 0.9130000000

681 2 windows install 0.3831768036 0.7180000000

682 2 windows install 0.3996810913 0.9230000000

683 2 windows install 0.3886520863 0.9090000000

684 2 windows install 0.3930339813 0.9280000000

685 2 windows install 0.3897500038 0.9410000000

686 2 windows install 0.3872950077 0.8930000000

687 2 windows install 0.3916780949 1.0810000000

688 2 windows install 0.3990821838 0.9520000000

689 2 windows install 0.3948898315 0.9190000000

690 2 windows install 0.4043509960 0.9380000000

691 2 windows install 0.3925690651 0.9210000000

692 2 windows install 0.3972198963 1.0130000000

693 2 windows install 0.3948929310 0.9410000000

694 2 windows install 0.3964531422 0.9460000000

695 2 windows install 0.3904659748 0.9380000000

696 2 windows install 0.3861818314 0.9210000000

697 2 windows install 0.3906159401 0.8930000000

698 2 windows install 0.3928079605 0.9330000000

699 2 windows install 0.3929181099 0.9720000000

700 2 windows install 0.3877868652 0.9080000000

701 2 windows install 0.3944458961 0.7370000000

702 2 windows install 0.3900849819 0.9160000000

703 2 windows install 0.3896040916 0.8880000000

704 2 windows install 0.1906218529 0.8430000000

705 2 windows install 0.1945629120 0.7400000000

706 2 windows install 0.3849110603 0.8850000000

707 2 windows install 0.1910879612 0.7480000000

708 2 windows install 0.3888540268 0.9340000000

709 2 windows install 0.3888959885 0.9960000000

710 2 windows install 0.3908510208 0.9110000000

711 2 windows install 0.3948848248 0.9320000000

712 2 windows install 0.3968751431 0.7530000000

713 2 windows install 0.4009110928 0.9430000000

Page 119: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXV

714 2 windows install 0.3808729649 0.9150000000

715 2 windows install 0.3933711052 0.9480000000

716 2 windows install 0.3889060020 0.7240000000

717 2 windows install 0.3966729641 0.9340000000

718 2 windows install 0.3826580048 0.9130000000

719 2 windows install 0.3982121944 0.9300000000

720 2 windows install 0.3926811218 0.9070000000

721 2 windows install 0.3987131119 0.9440000000

722 2 windows install 0.3983559608 0.9190000000

723 2 windows install 0.3941190243 0.7030000000

724 2 windows install 0.3947169781 0.9230000000

725 2 windows install 0.3900330067 0.9200000000

726 2 windows install 0.1952970028 0.7250000000

727 2 windows install 0.3827390671 0.8880000000

728 2 windows install 0.3947539330 0.9100000000

729 2 windows install 0.3893511295 0.9080000000

730 2 windows install 0.3908979893 0.9110000000

731 2 windows install 0.1890459061 0.7190000000

732 2 windows install 0.3961639404 0.9290000000

733 2 windows install 0.3932290077 0.9330000000

734 2 windows install 0.1881630421 0.7470000000

735 2 windows install 0.3911981583 0.9210000000

736 2 windows install 0.3930830956 0.9620000000

737 2 windows install 0.3986401558 1.0160000000

738 2 windows install 0.4060540199 0.7490000000

739 2 windows install 0.4048869610 1.0850000000

740 2 windows install 0.3909089565 0.9200000000

741 2 windows install 0.3962578773 1.0470000000

742 2 windows install 0.3917648792 1.0120000000

743 2 windows install 0.1891698837 0.6820000000

744 2 windows install 0.3907859325 0.9170000000

745 2 windows install 0.1911430359 0.7080000000

746 2 windows install 0.3930511475 0.9330000000

747 2 windows install 0.3963441849 0.9890000000

748 2 windows install 0.3882620335 0.9980000000

749 2 windows install 0.3967568874 1.0280000000

750 2 windows install 0.3895909786 0.9310000000

751 2 windows install 0.3984699249 0.9220000000

752 2 windows install 0.3850390911 0.6860000000

753 2 windows install 0.1869869232 0.5250000000

754 2 windows install 0.3969109058 0.9410000000

755 2 windows install 0.3826820850 0.8660000000

756 2 windows install 0.1920988560 0.7470000000

757 2 windows install 0.1929349899 0.7210000000

758 2 windows install 0.1892549992 0.6970000000

759 2 windows install 0.3822851181 0.9080000000

760 2 windows install 0.3939960003 0.9770000000

761 2 windows install 0.1873519421 0.5170000000

Page 120: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXVI

762 2 windows install 0.3977031708 0.9740000000

763 2 windows install 0.3932549953 0.9280000000

764 2 windows install 0.3930230141 0.9310000000

765 2 windows install 0.1949830055 0.8370000000

766 2 windows install 0.3950529099 0.9340000000

767 2 windows install 0.1929528713 0.7370000000

768 2 windows install 0.1912438869 0.7430000000

769 2 windows install 0.3909459114 0.9150000000

770 2 windows install 0.3893480301 1.0500000000

771 2 windows install 0.4048039913 1.0010000000

772 2 windows install 0.3852181435 0.8960000000

773 2 windows install 0.3885660172 0.9790000000

774 2 windows install 0.3868920803 0.7350000000

775 2 windows install 0.3931810856 0.9010000000

776 2 windows install 0.1869080067 0.7040000000

777 2 windows install 0.3927869797 0.9350000000

778 2 windows install 0.3928949833 0.9280000000

779 2 windows install 0.3899579048 0.8940000000

780 2 windows install 0.1971328259 0.5250000000

781 2 windows install 0.4067840576 0.9620000000

782 2 windows install 0.3948049545 0.9130000000

783 2 windows install 0.3886590004 0.9110000000

784 2 windows install 0.3917901516 1.0120000000

785 2 windows install 0.3945679665 0.9610000000

786 2 windows install 0.4110989571 0.9560000000

787 2 windows install 0.1875698566 0.8160000000

788 2 windows install 0.3971309662 0.9650000000

789 2 windows install 0.1870110035 0.5240000000

790 2 windows install 0.3938329220 0.9130000000

791 2 windows install 0.3836791515 0.7130000000

792 2 windows install 0.4012610912 0.9390000000

793 2 windows install 0.3906531334 0.9060000000

794 2 windows install 0.4113810062 0.9790000000

795 2 windows install 0.1887459755 0.4950000000

796 2 windows install 0.3964850903 0.9210000000

797 2 windows install 0.3881840706 0.9320000000

798 2 windows install 0.3907132149 1.0160000000

799 2 windows install 0.4014480114 0.9570000000

800 2 windows install 0.1968050003 0.7410000000

801 2 java get time 0.2072720528 0.8420000000

802 2 java get time 0.3975121975 0.9360000000

803 2 java get time 0.3894951344 0.9180000000

804 2 java get time 0.3975970745 0.9220000000

805 2 java get time 0.4014241695 0.9340000000

806 2 java get time 0.3970170021 0.9340000000

807 2 java get time 0.3911139965 1.0360000000

808 2 java get time 0.3912022114 0.9040000000

809 2 java get time 0.3890020847 0.9470000000

Page 121: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXVII

810 2 java get time 0.4000020027 0.9320000000

811 2 java get time 0.3931219578 0.9930000000

812 2 java get time 0.3909790516 0.9530000000

813 2 java get time 0.1919140816 0.8030000000

814 2 java get time 0.3930029869 0.9310000000

815 2 java get time 0.1894018650 0.7090000000

816 2 java get time 0.3930470943 0.9110000000

817 2 java get time 0.3922970295 0.8990000000

818 2 java get time 0.3976728916 1.0670000000

819 2 java get time 0.3949739933 0.9010000000

820 2 java get time 0.3942220211 0.9060000000

821 2 java get time 0.2062020302 0.6990000000

822 2 java get time 0.3840000629 0.9190000000

823 2 java get time 0.3913519382 0.9210000000

824 2 java get time 0.3908100128 0.9180000000

825 2 java get time 0.3923089504 0.7500000000

826 2 java get time 0.3895521164 0.8950000000

827 2 java get time 0.4004089832 0.9040000000

828 2 java get time 0.4015529156 0.8970000000

829 2 java get time 0.3910369873 0.9600000000

830 2 java get time 0.3873457909 0.9010000000

831 2 java get time 0.1990659237 0.7390000000

832 2 java get time 0.3952069283 0.9080000000

833 2 java get time 0.4054100513 0.9140000000

834 2 java get time 0.1953539848 0.7090000000

835 2 java get time 0.3861460686 0.9260000000

836 2 java get time 0.3932809830 0.9700000000

837 2 java get time 0.4055850506 0.9570000000

838 2 java get time 0.3911430836 0.7300000000

839 2 java get time 0.3913280964 0.9100000000

840 2 java get time 0.4096231461 0.9970000000

841 2 java get time 0.1910259724 0.7950000000

842 2 java get time 0.1945631504 0.7010000000

843 2 java get time 0.2012619972 0.8250000000

844 2 java get time 0.3812878132 0.9230000000

845 2 java get time 0.3923969269 0.9170000000

846 2 java get time 0.3952271938 0.9190000000

847 2 java get time 0.3897669315 0.9170000000

848 2 java get time 0.3927659988 0.9510000000

849 2 java get time 0.3986859322 0.9380000000

850 2 java get time 0.3899390697 0.8840000000

851 2 java get time 0.3812100887 0.9270000000

852 2 java get time 0.4042389393 1.0450000000

853 2 java get time 0.3965330124 0.9380000000

854 2 java get time 0.3975830078 0.9250000000

855 2 java get time 0.4003450871 0.7720000000

856 2 java get time 0.3831899166 0.9180000000

857 2 java get time 0.4006059170 0.9310000000

Page 122: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXVIII

858 2 java get time 0.3947589397 1.0340000000

859 2 java get time 0.1960129738 0.6960000000

860 2 java get time 0.3971102238 0.8920000000

861 2 java get time 0.3945901394 0.9220000000

862 2 java get time 0.3906629086 0.9850000000

863 2 java get time 0.3849332333 0.9000000000

864 2 java get time 0.2017719746 0.8590000000

865 2 java get time 0.3845949173 0.9170000000

866 2 java get time 0.3917000294 0.9390000000

867 2 java get time 0.1893179417 0.7480000000

868 2 java get time 0.3966619968 0.9220000000

869 2 java get time 0.4003281593 0.9880000000

870 2 java get time 0.3891999722 0.9450000000

871 2 java get time 0.3948531151 0.8080000000

872 2 java get time 0.3957529068 1.0000000000

873 2 java get time 0.1871509552 0.7090000000

874 2 java get time 0.3899641037 0.9020000000

875 2 java get time 0.1951849461 0.5250000000

876 2 java get time 0.4066381454 0.9320000000

877 2 java get time 0.3873441219 0.9640000000

878 2 java get time 0.3950319290 0.9190000000

879 2 java get time 0.1918611526 0.8300000000

880 2 java get time 0.1914660931 0.7090000000

881 2 java get time 0.3909139633 0.9130000000

882 2 java get time 0.3911919594 0.9520000000

883 2 java get time 0.3885149956 0.6740000000

884 2 java get time 0.4038469791 0.9320000000

885 2 java get time 0.3953971863 0.9670000000

886 2 java get time 0.3986718655 0.9240000000

887 2 java get time 0.3931310177 0.9130000000

888 2 java get time 0.3870749474 0.9600000000

889 2 java get time 0.3934769630 0.6990000000

890 2 java get time 0.3933579922 0.7350000000

891 2 java get time 0.3937640190 0.6920000000

892 2 java get time 0.3901820183 0.7130000000

893 2 java get time 0.1951689720 0.7030000000

894 2 java get time 0.4034249783 0.9170000000

895 2 java get time 0.3942201138 0.9270000000

896 2 java get time 0.3914270401 0.9690000000

897 2 java get time 0.1912698746 0.7460000000

898 2 java get time 0.4019060135 0.9300000000

899 2 java get time 0.2053010464 0.9130000000

900 2 java get time 0.3948960304 0.9220000000

901 2 java get time 0.3837599754 0.7320000000

902 2 java get time 0.3830609322 0.8770000000

903 2 java get time 0.3946921825 0.9210000000

904 2 java get time 0.1903529167 0.7150000000

905 2 java get time 0.3873331547 0.7040000000

Page 123: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

LXXXIX

906 2 java get time 0.3938088417 1.0220000000

907 2 java get time 0.3950049877 0.9420000000

908 2 java get time 0.3993539810 0.9220000000

909 2 java get time 0.3907511234 0.7000000000

910 2 java get time 0.3915209770 0.9990000000

911 2 java get time 0.3915529251 0.9210000000

912 2 java get time 0.3866181374 0.9130000000

913 2 java get time 0.1904590130 0.7090000000

914 2 java get time 0.3932669163 1.0040000000

915 2 java get time 0.3892750740 0.9020000000

916 2 java get time 0.3933069706 0.8940000000

917 2 java get time 0.3915090561 0.6930000000

918 2 java get time 0.4093990326 0.7060000000

919 2 java get time 0.3827841282 0.9110000000

920 2 java get time 0.3973770142 0.9270000000

921 2 java get time 0.3922569752 0.9150000000

922 2 java get time 0.1989989281 0.7310000000

923 2 java get time 0.3953030109 0.9530000000

924 2 java get time 0.1902499199 0.7360000000

925 2 java get time 0.3912518024 0.9200000000

926 2 java get time 0.3925931454 0.9860000000

927 2 java get time 0.3934199810 0.9370000000

928 2 java get time 0.1907348633 0.7230000000

929 2 java get time 0.4000620842 0.9720000000

930 2 java get time 0.3936159611 0.9150000000

931 2 java get time 0.3927619457 0.9350000000

932 2 java get time 0.3967700005 0.9970000000

933 2 java get time 0.3921890259 0.9120000000

934 2 java get time 0.3956849575 0.8920000000

935 2 java get time 0.4116380215 0.9430000000

936 2 java get time 0.3918859959 0.9180000000

937 2 java get time 0.4055099487 0.9180000000

938 2 java get time 0.4172759056 0.9430000000

939 2 java get time 0.3965950012 0.9170000000

940 2 java get time 0.3889150620 0.9360000000

941 2 java get time 0.3888449669 0.9340000000

942 2 java get time 0.3890969753 0.9410000000

943 2 java get time 0.3921048641 0.9160000000

944 2 java get time 0.3879461288 0.9830000000

945 2 java get time 0.3920860291 0.7520000000

946 2 java get time 0.1934850216 0.7290000000

947 2 java get time 0.3931629658 0.9580000000

948 2 java get time 0.3914949894 0.9140000000

949 2 java get time 0.3826861382 0.8820000000

950 2 java get time 0.3984868526 0.9390000000

951 2 java get time 0.3960690498 0.9520000000

952 2 java get time 0.3950588703 0.9080000000

953 2 java get time 0.4085400105 0.9440000000

Page 124: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XC

954 2 java get time 0.3934080601 0.7340000000

955 2 java get time 0.3978738785 0.9170000000

956 2 java get time 0.3871779442 0.8930000000

957 2 java get time 0.3909118176 0.9370000000

958 2 java get time 0.3975131512 0.9290000000

959 2 java get time 0.4033908844 0.9170000000

960 2 java get time 0.3951041698 1.0050000000

961 2 java get time 0.3964231014 0.9180000000

962 2 java get time 0.3934929371 0.9150000000

963 2 java get time 0.3940188885 0.8900000000

964 2 java get time 0.3890829086 0.9470000000

965 2 java get time 0.1995110512 0.7410000000

966 2 java get time 0.3870310783 0.9170000000

967 2 java get time 0.1919040680 0.7150000000

968 2 java get time 0.3944320679 0.9210000000

969 2 java get time 0.4009828568 0.9240000000

970 2 java get time 0.3965260983 0.7010000000

971 2 java get time 0.3948800564 1.0260000000

972 2 java get time 0.3892121315 0.9280000000

973 2 java get time 0.1911399364 0.7110000000

974 2 java get time 0.4031829834 0.9310000000

975 2 java get time 0.3930940628 0.7410000000

976 2 java get time 0.1979050636 0.7260000000

977 2 java get time 0.3911390305 0.9360000000

978 2 java get time 0.3912329674 0.7180000000

979 2 java get time 0.3931660652 0.9080000000

980 2 java get time 0.3933589458 0.8840000000

981 2 java get time 0.3961300850 1.0230000000

982 2 java get time 0.3993771076 1.0700000000

983 2 java get time 0.4144968987 1.0260000000

984 2 java get time 0.3968091011 0.7440000000

985 2 java get time 0.3979198933 1.1210000000

986 2 java get time 0.1933889389 0.7230000000

987 2 java get time 0.3954451084 0.9190000000

988 2 java get time 0.2063767910 0.7370000000

989 2 java get time 0.3931791782 0.7190000000

990 2 java get time 0.3930439949 0.9500000000

991 2 java get time 0.3929979801 0.9260000000

992 2 java get time 0.3906650543 0.9450000000

993 2 java get time 0.1921019554 0.7980000000

994 2 java get time 0.3890991211 0.9360000000

995 2 java get time 0.4095330238 0.9060000000

996 2 java get time 0.1907410622 0.7590000000

997 2 java get time 0.3892960548 0.8800000000

998 2 java get time 0.4020738602 0.9260000000

999 2 java get time 0.1963760853 0.5520000000

1000 2 java get time 0.4095680714 0.9580000000

1001 2 JavaScript Cookies 0.4112122059 0.9370000000

Page 125: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCI

1002 2 JavaScript Cookies 0.3937461376 0.7030000000

1003 2 JavaScript Cookies 0.3854291439 0.9100000000

1004 2 JavaScript Cookies 0.3979859352 0.7330000000

1005 2 JavaScript Cookies 0.3924520016 0.9430000000

1006 2 JavaScript Cookies 0.3939490318 0.9020000000

1007 2 JavaScript Cookies 0.3917019367 0.7050000000

1008 2 JavaScript Cookies 0.1967489719 0.6240000000

1009 2 JavaScript Cookies 0.3988969326 0.9680000000

1010 2 JavaScript Cookies 0.3890900612 0.9260000000

1011 2 JavaScript Cookies 0.3908801079 0.9890000000

1012 2 JavaScript Cookies 0.4041409492 0.9650000000

1013 2 JavaScript Cookies 0.3945610523 0.9050000000

1014 2 JavaScript Cookies 0.3925528526 0.8960000000

1015 2 JavaScript Cookies 0.4181091785 1.0080000000

1016 2 JavaScript Cookies 0.1891551018 0.7580000000

1017 2 JavaScript Cookies 0.3903050423 0.9250000000

1018 2 JavaScript Cookies 0.3971860409 0.9940000000

1019 2 JavaScript Cookies 0.3835709095 0.7140000000

1020 2 JavaScript Cookies 0.3927609921 1.0280000000

1021 2 JavaScript Cookies 0.4005961418 1.0670000000

1022 2 JavaScript Cookies 0.3927509785 0.9190000000

1023 2 JavaScript Cookies 0.3972690105 1.0630000000

1024 2 JavaScript Cookies 0.3945641518 0.9370000000

1025 2 JavaScript Cookies 0.3922090530 0.9320000000

1026 2 JavaScript Cookies 0.3946330547 0.9200000000

1027 2 JavaScript Cookies 0.3886640072 0.9270000000

1028 2 JavaScript Cookies 0.3979430199 0.9940000000

1029 2 JavaScript Cookies 0.3949651718 0.9960000000

1030 2 JavaScript Cookies 0.3946049213 0.9600000000

1031 2 JavaScript Cookies 0.4000310898 0.6980000000

1032 2 JavaScript Cookies 0.3923058510 0.7410000000

1033 2 JavaScript Cookies 0.1888580322 0.7030000000

1034 2 JavaScript Cookies 0.3954479694 0.8810000000

1035 2 JavaScript Cookies 0.3978791237 1.0480000000

1036 2 JavaScript Cookies 0.1912081242 0.7110000000

1037 2 JavaScript Cookies 0.3923931122 0.9460000000

1038 2 JavaScript Cookies 0.3938641548 0.9570000000

1039 2 JavaScript Cookies 0.3927299976 0.7000000000

1040 2 JavaScript Cookies 0.3990280628 0.9110000000

1041 2 JavaScript Cookies 0.3972208500 1.0330000000

1042 2 JavaScript Cookies 0.4008569717 1.0020000000

1043 2 JavaScript Cookies 0.3877310753 0.9470000000

1044 2 JavaScript Cookies 0.3825359344 0.9860000000

1045 2 JavaScript Cookies 0.3968989849 0.8900000000

1046 2 JavaScript Cookies 0.3982520103 0.9230000000

1047 2 JavaScript Cookies 0.3965258598 1.0040000000

1048 2 JavaScript Cookies 0.3894019127 0.9290000000

1049 2 JavaScript Cookies 0.3976659775 0.9360000000

Page 126: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCII

1050 2 JavaScript Cookies 0.3927199841 0.8880000000

1051 2 JavaScript Cookies 0.3926789761 0.9190000000

1052 2 JavaScript Cookies 0.3982441425 1.0440000000

1053 2 JavaScript Cookies 0.3919501305 0.9410000000

1054 2 JavaScript Cookies 0.4104270935 0.9530000000

1055 2 JavaScript Cookies 0.4077057838 0.9000000000

1056 2 JavaScript Cookies 0.3948101997 0.7450000000

1057 2 JavaScript Cookies 0.1946539879 0.7170000000

1058 2 JavaScript Cookies 0.3943369389 0.8030000000

1059 2 JavaScript Cookies 0.3981618881 0.7020000000

1060 2 JavaScript Cookies 0.1926209927 0.7050000000

1061 2 JavaScript Cookies 0.1987390518 0.5380000000

1062 2 JavaScript Cookies 0.1922919750 0.8250000000

1063 2 JavaScript Cookies 0.1914889812 0.7270000000

1064 2 JavaScript Cookies 0.1936361790 0.7520000000

1065 2 JavaScript Cookies 0.3922190666 1.0220000000

1066 2 JavaScript Cookies 0.1965119839 0.7030000000

1067 2 JavaScript Cookies 0.4062929153 1.0480000000

1068 2 JavaScript Cookies 0.3948171139 0.9250000000

1069 2 JavaScript Cookies 0.1921210289 0.7050000000

1070 2 JavaScript Cookies 0.3989298344 0.9100000000

1071 2 JavaScript Cookies 0.3896369934 0.9150000000

1072 2 JavaScript Cookies 0.1945741177 0.5310000000

1073 2 JavaScript Cookies 0.3896989822 0.9560000000

1074 2 JavaScript Cookies 0.3922188282 0.9250000000

1075 2 JavaScript Cookies 0.3957109451 0.9110000000

1076 2 JavaScript Cookies 0.2030601501 0.7490000000

1077 2 JavaScript Cookies 0.3971920013 1.0570000000

1078 2 JavaScript Cookies 0.2004430294 0.7340000000

1079 2 JavaScript Cookies 0.3954670429 0.9240000000

1080 2 JavaScript Cookies 0.1956608295 0.7420000000

1081 2 JavaScript Cookies 0.1906709671 0.7020000000

1082 2 JavaScript Cookies 0.3953118324 0.9060000000

1083 2 JavaScript Cookies 0.3906490803 1.0140000000

1084 2 JavaScript Cookies 0.4046440125 0.9370000000

1085 2 JavaScript Cookies 0.4161341190 0.7210000000

1086 2 JavaScript Cookies 0.3981199265 1.0330000000

1087 2 JavaScript Cookies 0.3901078701 0.7810000000

1088 2 JavaScript Cookies 0.3949201107 0.9080000000

1089 2 JavaScript Cookies 0.3991060257 0.9590000000

1090 2 JavaScript Cookies 0.3930101395 0.9250000000

1091 2 JavaScript Cookies 0.1929979324 0.7370000000

1092 2 JavaScript Cookies 0.3998792171 0.9610000000

1093 2 JavaScript Cookies 0.1922118664 0.7920000000

1094 2 JavaScript Cookies 0.3995199203 0.9560000000

1095 2 JavaScript Cookies 0.3823759556 0.9250000000

1096 2 JavaScript Cookies 0.3841059208 0.6880000000

1097 2 JavaScript Cookies 0.3886270523 0.9210000000

Page 127: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCIII

1098 2 JavaScript Cookies 0.3890769482 0.9250000000

1099 2 JavaScript Cookies 0.1886830330 0.6730000000

1100 2 JavaScript Cookies 0.1975629330 0.7510000000

1101 2 JavaScript Cookies 0.3991708755 0.9330000000

1102 2 JavaScript Cookies 0.3944039345 0.6930000000

1103 2 JavaScript Cookies 0.3916640282 0.9270000000

1104 2 JavaScript Cookies 0.3966729641 0.8930000000

1105 2 JavaScript Cookies 0.3996410370 0.9380000000

1106 2 JavaScript Cookies 0.3948249817 0.9090000000

1107 2 JavaScript Cookies 0.1970450878 0.7440000000

1108 2 JavaScript Cookies 0.3881611824 0.7720000000

1109 2 JavaScript Cookies 0.3903570175 0.9090000000

1110 2 JavaScript Cookies 0.3938941956 0.8920000000

1111 2 JavaScript Cookies 0.3928749561 0.7450000000

1112 2 JavaScript Cookies 0.3997819424 0.9030000000

1113 2 JavaScript Cookies 0.3880510330 0.7110000000

1114 2 JavaScript Cookies 0.3824660778 1.0300000000

1115 2 JavaScript Cookies 0.3949661255 0.7060000000

1116 2 JavaScript Cookies 0.3868279457 0.9060000000

1117 2 JavaScript Cookies 0.1860268116 0.7370000000

1118 2 JavaScript Cookies 0.3912320137 0.6780000000

1119 2 JavaScript Cookies 0.3977248669 0.9150000000

1120 2 JavaScript Cookies 0.3962700367 0.7060000000

1121 2 JavaScript Cookies 0.3901760578 0.9170000000

1122 2 JavaScript Cookies 0.3947489262 0.9530000000

1123 2 JavaScript Cookies 0.1902611256 0.5080000000

1124 2 JavaScript Cookies 0.1999430656 0.7000000000

1125 2 JavaScript Cookies 0.3949508667 0.9150000000

1126 2 JavaScript Cookies 0.3988118172 0.7040000000

1127 2 JavaScript Cookies 0.3797800541 0.6950000000

1128 2 JavaScript Cookies 0.4057409763 0.7230000000

1129 2 JavaScript Cookies 0.4075908661 0.9360000000

1130 2 JavaScript Cookies 0.3952169418 0.9330000000

1131 2 JavaScript Cookies 0.3924429417 0.9130000000

1132 2 JavaScript Cookies 0.3901510239 0.9050000000

1133 2 JavaScript Cookies 0.3965029716 0.9380000000

1134 2 JavaScript Cookies 0.3969738483 0.9290000000

1135 2 JavaScript Cookies 0.3969490528 0.9110000000

1136 2 JavaScript Cookies 0.3790998459 0.8890000000

1137 2 JavaScript Cookies 0.1911451817 0.7240000000

1138 2 JavaScript Cookies 0.1964108944 0.7160000000

1139 2 JavaScript Cookies 0.1885108948 0.6890000000

1140 2 JavaScript Cookies 0.3965399265 0.7660000000

1141 2 JavaScript Cookies 0.4076180458 0.9820000000

1142 2 JavaScript Cookies 0.4002890587 0.9390000000

1143 2 JavaScript Cookies 0.3935301304 0.9090000000

1144 2 JavaScript Cookies 0.4026191235 1.0830000000

1145 2 JavaScript Cookies 0.3954420090 0.9170000000

Page 128: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCIV

1146 2 JavaScript Cookies 0.3949730396 0.9250000000

1147 2 JavaScript Cookies 0.3899469376 0.7110000000

1148 2 JavaScript Cookies 0.3921451569 1.0000000000

1149 2 JavaScript Cookies 0.3886477947 0.9370000000

1150 2 JavaScript Cookies 0.3935980797 0.9010000000

1151 2 JavaScript Cookies 0.3944368362 0.7670000000

1152 2 JavaScript Cookies 0.3890218735 0.7070000000

1153 2 JavaScript Cookies 0.3976819515 0.9440000000

1154 2 JavaScript Cookies 0.3900120258 0.7400000000

1155 2 JavaScript Cookies 0.3882000446 0.9290000000

1156 2 JavaScript Cookies 0.3866550922 0.9170000000

1157 2 JavaScript Cookies 0.3912739754 0.8900000000

1158 2 JavaScript Cookies 0.3905398846 1.0070000000

1159 2 JavaScript Cookies 0.4068930149 0.8920000000

1160 2 JavaScript Cookies 0.1912591457 0.4930000000

1161 2 JavaScript Cookies 0.1949231625 0.4980000000

1162 2 JavaScript Cookies 0.3941509724 0.9100000000

1163 2 JavaScript Cookies 0.1924579144 0.7040000000

1164 2 JavaScript Cookies 0.3984589577 0.8670000000

1165 2 JavaScript Cookies 0.4001691341 0.9840000000

1166 2 JavaScript Cookies 0.3890211582 0.8780000000

1167 2 JavaScript Cookies 0.3954589367 0.7050000000

1168 2 JavaScript Cookies 0.3944220543 0.7140000000

1169 2 JavaScript Cookies 0.1911690235 0.7160000000

1170 2 JavaScript Cookies 0.3890848160 0.9290000000

1171 2 JavaScript Cookies 0.4073848724 0.7220000000

1172 2 JavaScript Cookies 0.3910808563 0.8900000000

1173 2 JavaScript Cookies 0.2054572105 0.9430000000

1174 2 JavaScript Cookies 0.3922891617 0.9160000000

1175 2 JavaScript Cookies 0.3905198574 0.9300000000

1176 2 JavaScript Cookies 0.1966300011 0.4870000000

1177 2 JavaScript Cookies 0.3967280388 0.7350000000

1178 2 JavaScript Cookies 0.1916460991 0.7180000000

1179 2 JavaScript Cookies 0.4023611546 0.7100000000

1180 2 JavaScript Cookies 0.1903319359 0.7630000000

1181 2 JavaScript Cookies 0.3927009106 0.9700000000

1182 2 JavaScript Cookies 0.1942818165 0.7590000000

1183 2 JavaScript Cookies 0.3916909695 0.9280000000

1184 2 JavaScript Cookies 0.3926560879 1.0800000000

1185 2 JavaScript Cookies 0.4101190567 0.9530000000

1186 2 JavaScript Cookies 0.3931300640 0.9040000000

1187 2 JavaScript Cookies 0.3966739178 0.9180000000

1188 2 JavaScript Cookies 0.4003331661 0.9020000000

1189 2 JavaScript Cookies 0.3928630352 0.9050000000

1190 2 JavaScript Cookies 0.4010579586 0.9260000000

1191 2 JavaScript Cookies 0.3935050964 0.9220000000

1192 2 JavaScript Cookies 0.3931128979 0.9430000000

1193 2 JavaScript Cookies 0.3917400837 0.9320000000

Page 129: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCV

1194 2 JavaScript Cookies 0.3916201591 0.9540000000

1195 2 JavaScript Cookies 0.3905940056 0.9080000000

1196 2 JavaScript Cookies 0.3954410553 1.0380000000

1197 2 JavaScript Cookies 0.3904819489 0.8840000000

1198 2 JavaScript Cookies 0.4033238888 1.1080000000

1199 2 JavaScript Cookies 0.3952960968 0.9000000000

1200 2 JavaScript Cookies 0.1965739727 0.7120000000

1201 3 windows install 0.3524281979 0.9300000000

1202 3 windows install 0.3548169136 0.8890000000

1203 3 windows install 0.3511800766 0.8580000000

1204 3 windows install 0.3550899029 0.8750000000

1205 3 windows install 0.3564839363 0.9020000000

1206 3 windows install 0.3546528816 0.9910000000

1207 3 windows install 0.3568751812 0.7160000000

1208 3 windows install 0.3618330956 0.8990000000

1209 3 windows install 0.3554689884 0.9140000000

1210 3 windows install 0.3528959751 0.6720000000

1211 3 windows install 0.3531589508 0.9910000000

1212 3 windows install 0.3535449505 0.6590000000

1213 3 windows install 0.3533971310 0.8450000000

1214 3 windows install 0.3543407917 0.6460000000

1215 3 windows install 0.3494601250 0.9000000000

1216 3 windows install 0.3556029797 0.9120000000

1217 3 windows install 0.3431060314 0.8440000000

1218 3 windows install 0.3534111977 0.8840000000

1219 3 windows install 0.3508019447 0.8520000000

1220 3 windows install 0.3470640182 0.6860000000

1221 3 windows install 0.1567180157 0.7740000000

1222 3 windows install 0.3538658619 0.9060000000

1223 3 windows install 0.3719420433 0.8560000000

1224 3 windows install 0.3516759872 0.8460000000

1225 3 windows install 0.3572199345 0.8820000000

1226 3 windows install 0.3562850952 0.8710000000

1227 3 windows install 0.3512251377 0.9010000000

1228 3 windows install 0.3545179367 0.7800000000

1229 3 windows install 0.3591399193 0.8860000000

1230 3 windows install 0.3566019535 0.8800000000

1231 3 windows install 0.3513710499 0.6450000000

1232 3 windows install 0.3571619987 0.9110000000

1233 3 windows install 0.3536190987 0.8870000000

1234 3 windows install 0.3659281731 0.8630000000

1235 3 windows install 0.1551480293 0.6760000000

1236 3 windows install 0.3535380363 0.8950000000

1237 3 windows install 0.3538091183 0.6860000000

1238 3 windows install 0.3713660240 1.0200000000

1239 3 windows install 0.1543509960 0.6950000000

1240 3 windows install 0.3528621197 0.8870000000

1241 3 windows install 0.3588471413 0.6970000000

Page 130: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCVI

1242 3 windows install 0.3543579578 0.8760000000

1243 3 windows install 0.1563520432 0.6520000000

1244 3 windows install 0.3530328274 0.8760000000

1245 3 windows install 0.3486399651 0.8820000000

1246 3 windows install 0.3611760139 0.8880000000

1247 3 windows install 0.3489649296 0.8700000000

1248 3 windows install 0.1549179554 0.7880000000

1249 3 windows install 0.3482549191 0.6390000000

1250 3 windows install 0.3651850224 0.9000000000

1251 3 windows install 0.3557789326 1.0400000000

1252 3 windows install 0.3661000729 0.6790000000

1253 3 windows install 0.3529708385 0.8540000000

1254 3 windows install 0.3544769287 0.8690000000

1255 3 windows install 0.3649902344 0.9360000000

1256 3 windows install 0.3529798985 0.9140000000

1257 3 windows install 0.3596730232 0.8930000000

1258 3 windows install 0.3566339016 0.9140000000

1259 3 windows install 0.3613209724 0.8700000000

1260 3 windows install 0.1557638645 0.8120000000

1261 3 windows install 0.1575238705 0.6660000000

1262 3 windows install 0.3521800041 0.6630000000

1263 3 windows install 0.3631958961 0.6720000000

1264 3 windows install 0.3745272160 0.8910000000

1265 3 windows install 0.3445849419 0.6610000000

1266 3 windows install 0.3544180393 0.8560000000

1267 3 windows install 0.3555490971 1.0420000000

1268 3 windows install 0.1508479118 0.6540000000

1269 3 windows install 0.3536391258 0.8780000000

1270 3 windows install 0.3497190475 0.9050000000

1271 3 windows install 0.3534328938 0.9050000000

1272 3 windows install 0.3635699749 0.8920000000

1273 3 windows install 0.3532049656 0.8740000000

1274 3 windows install 0.3569779396 0.8880000000

1275 3 windows install 0.3507230282 0.9590000000

1276 3 windows install 0.3532900810 0.8690000000

1277 3 windows install 0.3601930141 0.8710000000

1278 3 windows install 0.3548719883 0.7920000000

1279 3 windows install 0.3495030403 0.9250000000

1280 3 windows install 0.3551011086 0.9160000000

1281 3 windows install 0.3767268658 0.9190000000

1282 3 windows install 0.3510990143 0.6960000000

1283 3 windows install 0.1648030281 0.6850000000

1284 3 windows install 0.3567829132 0.8830000000

1285 3 windows install 0.3547401428 0.9500000000

1286 3 windows install 0.3547329903 0.8950000000

1287 3 windows install 0.3520059586 0.8510000000

1288 3 windows install 0.3554780483 0.8940000000

1289 3 windows install 0.3504819870 0.8860000000

Page 131: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCVII

1290 3 windows install 0.3663439751 0.9040000000

1291 3 windows install 0.1595270634 0.6640000000

1292 3 windows install 0.3568868637 0.8700000000

1293 3 windows install 0.1526408195 0.6660000000

1294 3 windows install 0.3550379276 0.9940000000

1295 3 windows install 0.3450388908 0.6370000000

1296 3 windows install 0.3543980122 0.7190000000

1297 3 windows install 0.1512567997 0.6820000000

1298 3 windows install 0.3494389057 0.9810000000

1299 3 windows install 0.3704218864 0.8860000000

1300 3 windows install 0.3595640659 0.9360000000

1301 3 windows install 0.3565859795 0.8980000000

1302 3 windows install 0.3537640572 0.8650000000

1303 3 windows install 0.3452301025 1.0120000000

1304 3 windows install 0.3591959476 0.9110000000

1305 3 windows install 0.3536858559 0.8720000000

1306 3 windows install 0.3546671867 0.9210000000

1307 3 windows install 0.3626999855 0.6780000000

1308 3 windows install 0.3637678623 0.6580000000

1309 3 windows install 0.3587779999 0.8900000000

1310 3 windows install 0.3526868820 0.9660000000

1311 3 windows install 0.3551430702 0.9210000000

1312 3 windows install 0.3470849991 0.8680000000

1313 3 windows install 0.3549270630 0.9760000000

1314 3 windows install 0.3533160686 0.8920000000

1315 3 windows install 0.3573708534 0.7280000000

1316 3 windows install 0.3529570103 0.8900000000

1317 3 windows install 0.3503389359 0.8640000000

1318 3 windows install 0.3533639908 0.8730000000

1319 3 windows install 0.3532929420 0.8610000000

1320 3 windows install 0.1566569805 0.6600000000

1321 3 windows install 0.3554670811 0.9580000000

1322 3 windows install 0.3550658226 1.0110000000

1323 3 windows install 0.3589770794 0.7000000000

1324 3 windows install 0.3546659946 0.8970000000

1325 3 windows install 0.3545529842 0.6820000000

1326 3 windows install 0.3513050079 0.7410000000

1327 3 windows install 0.3531551361 0.8980000000

1328 3 windows install 0.3494350910 0.8530000000

1329 3 windows install 0.3454051018 0.8900000000

1330 3 windows install 0.3526570797 0.8990000000

1331 3 windows install 0.3683478832 0.8870000000

1332 3 windows install 0.3572318554 0.8680000000

1333 3 windows install 0.3642899990 0.9260000000

1334 3 windows install 0.3574328423 0.9060000000

1335 3 windows install 0.3501679897 1.0540000000

1336 3 windows install 0.1606011391 0.7180000000

1337 3 windows install 0.3590970039 0.8680000000

Page 132: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCVIII

1338 3 windows install 0.3557579517 0.8730000000

1339 3 windows install 0.3509490490 0.9000000000

1340 3 windows install 0.3493928909 0.8820000000

1341 3 windows install 0.3529548645 0.9190000000

1342 3 windows install 0.3561339378 1.0800000000

1343 3 windows install 0.3546731472 0.8780000000

1344 3 windows install 0.3613479137 0.7020000000

1345 3 windows install 0.3534719944 0.6710000000

1346 3 windows install 0.3512217999 0.8930000000

1347 3 windows install 0.1535460949 0.6630000000

1348 3 windows install 0.1511728764 0.4720000000

1349 3 windows install 0.3474709988 0.6870000000

1350 3 windows install 0.3566191196 0.8510000000

1351 3 windows install 0.3453109264 0.9250000000

1352 3 windows install 0.3560609818 0.9080000000

1353 3 windows install 0.1543219090 0.6590000000

1354 3 windows install 0.1525280476 0.7100000000

1355 3 windows install 0.3588998318 0.8620000000

1356 3 windows install 0.1510858536 0.7710000000

1357 3 windows install 0.3574318886 0.9020000000

1358 3 windows install 0.3645269871 0.7230000000

1359 3 windows install 0.1568870544 0.7090000000

1360 3 windows install 0.3507099152 0.6540000000

1361 3 windows install 0.3514449596 0.8810000000

1362 3 windows install 0.3643820286 0.6680000000

1363 3 windows install 0.3520309925 0.9510000000

1364 3 windows install 0.3535728455 0.8780000000

1365 3 windows install 0.3526759148 0.8790000000

1366 3 windows install 0.3504149914 0.8880000000

1367 3 windows install 0.1553549767 0.6710000000

1368 3 windows install 0.3526589870 0.6760000000

1369 3 windows install 0.3510708809 0.6680000000

1370 3 windows install 0.3551077843 0.8760000000

1371 3 windows install 0.3548130989 0.8720000000

1372 3 windows install 0.3546049595 0.7870000000

1373 3 windows install 0.3554158211 0.8840000000

1374 3 windows install 0.3569459915 0.8630000000

1375 3 windows install 0.1511759758 0.6740000000

1376 3 windows install 0.1555359364 0.7090000000

1377 3 windows install 0.3568348885 0.8930000000

1378 3 windows install 0.3570380211 0.8580000000

1379 3 windows install 0.3505089283 0.8890000000

1380 3 windows install 0.3526198864 0.9430000000

1381 3 windows install 0.1599469185 0.6490000000

1382 3 windows install 0.3504869938 0.8810000000

1383 3 windows install 0.3550579548 0.8880000000

1384 3 windows install 0.3443820477 0.8600000000

1385 3 windows install 0.1543629169 0.6760000000

Page 133: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

XCIX

1386 3 windows install 0.3529789448 0.8960000000

1387 3 windows install 0.3594901562 0.9610000000

1388 3 windows install 0.3554282188 0.8890000000

1389 3 windows install 0.3553390503 0.6810000000

1390 3 windows install 0.1604890823 0.6760000000

1391 3 windows install 0.3549261093 0.6800000000

1392 3 windows install 0.3527350426 0.9130000000

1393 3 windows install 0.1581130028 0.7010000000

1394 3 windows install 0.3606479168 0.8860000000

1395 3 windows install 0.3647379875 0.8920000000

1396 3 windows install 0.3548557758 0.8540000000

1397 3 windows install 0.3468949795 0.8810000000

1398 3 windows install 0.3570950031 0.6540000000

1399 3 windows install 0.3533821106 0.8610000000

1400 3 windows install 0.3552989960 0.9370000000

1401 3 java get time 0.3709199429 0.9300000000

1402 3 java get time 0.3532400131 0.9590000000

1403 3 java get time 0.1505310535 0.7040000000

1404 3 java get time 0.3605458736 0.8830000000

1405 3 java get time 0.1492838860 0.7290000000

1406 3 java get time 0.3476660252 0.6550000000

1407 3 java get time 0.1514079571 0.6770000000

1408 3 java get time 0.1538321972 0.7430000000

1409 3 java get time 0.1514270306 0.7110000000

1410 3 java get time 0.3470969200 0.9350000000

1411 3 java get time 0.1547429562 0.5250000000

1412 3 java get time 0.3584938049 0.9280000000

1413 3 java get time 0.3595979214 0.8910000000

1414 3 java get time 0.3597030640 0.9110000000

1415 3 java get time 0.3547489643 1.0620000000

1416 3 java get time 0.3537039757 0.8900000000

1417 3 java get time 0.1525111198 0.6390000000

1418 3 java get time 0.1492321491 0.7180000000

1419 3 java get time 0.3541140556 0.8860000000

1420 3 java get time 0.3509609699 0.8770000000

1421 3 java get time 0.3520159721 0.9010000000

1422 3 java get time 0.3566620350 0.8620000000

1423 3 java get time 0.3447768688 0.9000000000

1424 3 java get time 0.1520471573 0.6780000000

1425 3 java get time 0.1674809456 0.7130000000

1426 3 java get time 0.1498751640 0.6790000000

1427 3 java get time 0.3563468456 0.9120000000

1428 3 java get time 0.3490967751 0.9190000000

1429 3 java get time 0.1635019779 0.7030000000

1430 3 java get time 0.3500509262 0.9570000000

1431 3 java get time 0.3513150215 0.8490000000

1432 3 java get time 0.1562559605 0.7370000000

1433 3 java get time 0.3507771492 0.8620000000

Page 134: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

C

1434 3 java get time 0.3546319008 0.6720000000

1435 3 java get time 0.3650820255 1.0050000000

1436 3 java get time 0.3533639908 0.9630000000

1437 3 java get time 0.3541040421 0.9150000000

1438 3 java get time 0.3539860249 0.9520000000

1439 3 java get time 0.3605940342 0.9520000000

1440 3 java get time 0.3505752087 0.8600000000

1441 3 java get time 0.3526351452 0.8820000000

1442 3 java get time 0.3420839310 0.7000000000

1443 3 java get time 0.3474588394 0.7030000000

1444 3 java get time 0.3525810242 0.8730000000

1445 3 java get time 0.1528611183 0.6680000000

1446 3 java get time 0.3561730385 0.8630000000

1447 3 java get time 0.3756580353 0.9090000000

1448 3 java get time 0.3563218117 0.8690000000

1449 3 java get time 0.3741497993 0.7660000000

1450 3 java get time 0.1528971195 0.6870000000

1451 3 java get time 0.3538088799 0.6870000000

1452 3 java get time 0.3692851067 0.8750000000

1453 3 java get time 0.3501369953 0.6780000000

1454 3 java get time 0.3523859978 0.8790000000

1455 3 java get time 0.1506869793 0.5400000000

1456 3 java get time 0.3613028526 0.9090000000

1457 3 java get time 0.3513808250 0.9360000000

1458 3 java get time 0.3547389507 0.9950000000

1459 3 java get time 0.3550848961 0.8950000000

1460 3 java get time 0.3741440773 0.9120000000

1461 3 java get time 0.3534920216 0.8860000000

1462 3 java get time 0.3528311253 0.7680000000

1463 3 java get time 0.3550069332 0.9070000000

1464 3 java get time 0.3511300087 0.9490000000

1465 3 java get time 0.3635840416 0.8880000000

1466 3 java get time 0.3545451164 0.9300000000

1467 3 java get time 0.3529701233 0.8880000000

1468 3 java get time 0.3533890247 0.9090000000

1469 3 java get time 0.3511741161 0.8560000000

1470 3 java get time 0.3646740913 0.8810000000

1471 3 java get time 0.3556160927 0.8600000000

1472 3 java get time 0.3552451134 0.6440000000

1473 3 java get time 0.1482160091 0.6670000000

1474 3 java get time 0.3602688313 1.0060000000

1475 3 java get time 0.1511468887 0.7040000000

1476 3 java get time 0.3548901081 0.8960000000

1477 3 java get time 0.3684139252 0.9130000000

1478 3 java get time 0.3488900661 0.6800000000

1479 3 java get time 0.3507211208 0.8770000000

1480 3 java get time 0.3532969952 0.8620000000

1481 3 java get time 0.1532208920 0.7050000000

Page 135: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CI

1482 3 java get time 0.3634719849 0.8780000000

1483 3 java get time 0.3543698788 0.8900000000

1484 3 java get time 0.3516108990 0.8950000000

1485 3 java get time 0.3515551090 0.8630000000

1486 3 java get time 0.3652801514 0.8870000000

1487 3 java get time 0.1546130180 0.6470000000

1488 3 java get time 0.3663501740 0.8670000000

1489 3 java get time 0.1528770924 0.6620000000

1490 3 java get time 0.3579788208 0.6930000000

1491 3 java get time 0.3531961441 0.8950000000

1492 3 java get time 0.3508348465 0.6630000000

1493 3 java get time 0.3575530052 1.0230000000

1494 3 java get time 0.1523981094 0.5810000000

1495 3 java get time 0.3553597927 0.9010000000

1496 3 java get time 0.1537928581 0.7170000000

1497 3 java get time 0.3516268730 0.8990000000

1498 3 java get time 0.3629388809 0.9060000000

1499 3 java get time 0.3609049320 0.8850000000

1500 3 java get time 0.3450598717 1.0080000000

1501 3 java get time 0.3504190445 0.9010000000

1502 3 java get time 0.1499118805 0.6900000000

1503 3 java get time 0.3510570526 0.8690000000

1504 3 java get time 0.3594560623 0.8760000000

1505 3 java get time 0.3543920517 0.7400000000

1506 3 java get time 0.3538711071 0.8760000000

1507 3 java get time 0.3474221230 0.8770000000

1508 3 java get time 0.3542940617 0.8910000000

1509 3 java get time 0.3700120449 0.8720000000

1510 3 java get time 0.3588838577 0.8560000000

1511 3 java get time 0.3521480560 0.6830000000

1512 3 java get time 0.3574128151 0.9870000000

1513 3 java get time 0.1531939507 0.6800000000

1514 3 java get time 0.3468768597 0.8470000000

1515 3 java get time 0.3548381329 0.8690000000

1516 3 java get time 0.3528110981 0.6410000000

1517 3 java get time 0.3551089764 0.6990000000

1518 3 java get time 0.3543190956 0.7680000000

1519 3 java get time 0.3519980907 0.9000000000

1520 3 java get time 0.1505379677 0.6840000000

1521 3 java get time 0.3530859947 0.8520000000

1522 3 java get time 0.1503329277 0.6800000000

1523 3 java get time 0.3626079559 0.8670000000

1524 3 java get time 0.3557147980 0.9910000000

1525 3 java get time 0.3562090397 0.9210000000

1526 3 java get time 0.3589310646 0.8630000000

1527 3 java get time 0.3564629555 0.8780000000

1528 3 java get time 0.1499359608 0.6640000000

1529 3 java get time 0.1523418427 0.6640000000

Page 136: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CII

1530 3 java get time 0.3697438240 1.0410000000

1531 3 java get time 0.1527640820 0.6970000000

1532 3 java get time 0.3508570194 0.8580000000

1533 3 java get time 0.3492450714 0.8840000000

1534 3 java get time 0.3514070511 0.9000000000

1535 3 java get time 0.3563139439 0.9600000000

1536 3 java get time 0.1529660225 0.4680000000

1537 3 java get time 0.3573670387 0.6960000000

1538 3 java get time 0.3505520821 0.8940000000

1539 3 java get time 0.3487591743 0.8820000000

1540 3 java get time 0.3491971493 1.0000000000

1541 3 java get time 0.3484640121 0.8490000000

1542 3 java get time 0.3529920578 0.8820000000

1543 3 java get time 0.3624289036 0.9100000000

1544 3 java get time 0.3509120941 0.8890000000

1545 3 java get time 0.1496911049 0.6810000000

1546 3 java get time 0.3508961201 0.9900000000

1547 3 java get time 0.3729159832 1.0310000000

1548 3 java get time 0.3543708324 0.8490000000

1549 3 java get time 0.3503129482 0.8640000000

1550 3 java get time 0.1560101509 0.6340000000

1551 3 java get time 0.3537609577 0.7850000000

1552 3 java get time 0.3614840508 0.8940000000

1553 3 java get time 0.3531208038 0.8940000000

1554 3 java get time 0.3541388512 0.8940000000

1555 3 java get time 0.3513820171 0.7010000000

1556 3 java get time 0.3725419044 0.9150000000

1557 3 java get time 0.3551709652 0.8940000000

1558 3 java get time 0.3650660515 0.8540000000

1559 3 java get time 0.1601040363 0.5030000000

1560 3 java get time 0.3567788601 0.8670000000

1561 3 java get time 0.3512990475 0.8730000000

1562 3 java get time 0.3527820110 0.9130000000

1563 3 java get time 0.3514790535 0.8840000000

1564 3 java get time 0.3566880226 0.9280000000

1565 3 java get time 0.3509578705 0.9180000000

1566 3 java get time 0.3449940681 0.8900000000

1567 3 java get time 0.3539550304 0.8880000000

1568 3 java get time 0.3529400826 0.8800000000

1569 3 java get time 0.3495118618 0.8440000000

1570 3 java get time 0.3606441021 0.8850000000

1571 3 java get time 0.3509249687 0.8970000000

1572 3 java get time 0.3549470901 0.9190000000

1573 3 java get time 0.3548829556 0.8920000000

1574 3 java get time 0.3453240395 0.7900000000

1575 3 java get time 0.3523271084 0.8830000000

1576 3 java get time 0.3523809910 0.8630000000

1577 3 java get time 0.1493039131 0.6850000000

Page 137: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CIII

1578 3 java get time 0.3549759388 0.8910000000

1579 3 java get time 0.3537931442 0.8730000000

1580 3 java get time 0.1540679932 0.6840000000

1581 3 java get time 0.3516509533 0.8600000000

1582 3 java get time 0.3535189629 0.6890000000

1583 3 java get time 0.3581349850 0.8770000000

1584 3 java get time 0.3543250561 0.6830000000

1585 3 java get time 0.1530470848 0.6730000000

1586 3 java get time 0.3569810390 0.9450000000

1587 3 java get time 0.1533420086 0.4420000000

1588 3 java get time 0.3477890491 0.8630000000

1589 3 java get time 0.3564271927 0.8600000000

1590 3 java get time 0.3538219929 0.8610000000

1591 3 java get time 0.3536639214 0.8820000000

1592 3 java get time 0.3560619354 0.8930000000

1593 3 java get time 0.3577218056 1.0660000000

1594 3 java get time 0.3513779640 0.8570000000

1595 3 java get time 0.3530790806 0.9270000000

1596 3 java get time 0.3685021400 0.9160000000

1597 3 java get time 0.3530530930 0.8820000000

1598 3 java get time 0.3526279926 0.8600000000

1599 3 java get time 0.3611750603 0.8590000000

1600 3 java get time 0.1506628990 0.6660000000

1601 3 JavaScript Cookies 0.1528930664 0.6570000000

1602 3 JavaScript Cookies 0.3702919483 0.8990000000

1603 3 JavaScript Cookies 0.1511130333 0.7030000000

1604 3 JavaScript Cookies 0.1503999233 0.6610000000

1605 3 JavaScript Cookies 0.3525731564 0.9180000000

1606 3 JavaScript Cookies 0.3606810570 0.8910000000

1607 3 JavaScript Cookies 0.3506779671 0.9110000000

1608 3 JavaScript Cookies 0.3427188396 0.8920000000

1609 3 JavaScript Cookies 0.3441569805 0.8470000000

1610 3 JavaScript Cookies 0.3445799351 0.8850000000

1611 3 JavaScript Cookies 0.3649420738 0.8780000000

1612 3 JavaScript Cookies 0.3450038433 0.8930000000

1613 3 JavaScript Cookies 0.3537521362 0.6640000000

1614 3 JavaScript Cookies 0.3568220139 0.6840000000

1615 3 JavaScript Cookies 0.3527419567 0.8590000000

1616 3 JavaScript Cookies 0.3587579727 0.8680000000

1617 3 JavaScript Cookies 0.1513240337 0.7040000000

1618 3 JavaScript Cookies 0.3546879292 0.8740000000

1619 3 JavaScript Cookies 0.3504810333 0.8930000000

1620 3 JavaScript Cookies 0.3524301052 0.8960000000

1621 3 JavaScript Cookies 0.3569288254 0.8610000000

1622 3 JavaScript Cookies 0.1519939899 0.5310000000

1623 3 JavaScript Cookies 0.3540031910 0.8940000000

1624 3 JavaScript Cookies 0.3525288105 0.8680000000

1625 3 JavaScript Cookies 0.3532149792 0.6730000000

Page 138: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CIV

1626 3 JavaScript Cookies 0.3525929451 0.8900000000

1627 3 JavaScript Cookies 0.3562409878 0.6980000000

1628 3 JavaScript Cookies 0.3431408405 0.8250000000

1629 3 JavaScript Cookies 0.3531899452 0.6830000000

1630 3 JavaScript Cookies 0.3691110611 0.8610000000

1631 3 JavaScript Cookies 0.3546791077 0.7060000000

1632 3 JavaScript Cookies 0.3502650261 0.6510000000

1633 3 JavaScript Cookies 0.3522808552 0.8810000000

1634 3 JavaScript Cookies 0.3550229073 0.9800000000

1635 3 JavaScript Cookies 0.3514668941 0.8970000000

1636 3 JavaScript Cookies 0.3505549431 0.9050000000

1637 3 JavaScript Cookies 0.3572900295 0.8960000000

1638 3 JavaScript Cookies 0.3489348888 0.8600000000

1639 3 JavaScript Cookies 0.3600931168 0.8810000000

1640 3 JavaScript Cookies 0.1708390713 0.6590000000

1641 3 JavaScript Cookies 0.3524000645 0.8440000000

1642 3 JavaScript Cookies 0.3500938416 0.8950000000

1643 3 JavaScript Cookies 0.3513219357 0.9130000000

1644 3 JavaScript Cookies 0.3505749702 0.8480000000

1645 3 JavaScript Cookies 0.3675479889 0.8830000000

1646 3 JavaScript Cookies 0.3566870689 0.7360000000

1647 3 JavaScript Cookies 0.3506479263 0.8830000000

1648 3 JavaScript Cookies 0.3525669575 0.8820000000

1649 3 JavaScript Cookies 0.3527309895 0.9090000000

1650 3 JavaScript Cookies 0.3529319763 0.8490000000

1651 3 JavaScript Cookies 0.1513090134 0.6710000000

1652 3 JavaScript Cookies 0.3527820110 0.9080000000

1653 3 JavaScript Cookies 0.3630118370 0.8930000000

1654 3 JavaScript Cookies 0.1544289589 0.7020000000

1655 3 JavaScript Cookies 0.1503078938 0.8520000000

1656 3 JavaScript Cookies 0.3594629765 0.9020000000

1657 3 JavaScript Cookies 0.3664970398 0.9180000000

1658 3 JavaScript Cookies 0.1523211002 0.7230000000

1659 3 JavaScript Cookies 0.1508390903 0.8570000000

1660 3 JavaScript Cookies 0.3528680801 0.6930000000

1661 3 JavaScript Cookies 0.3529698849 0.8610000000

1662 3 JavaScript Cookies 0.3535211086 0.9480000000

1663 3 JavaScript Cookies 0.3534638882 0.8680000000

1664 3 JavaScript Cookies 0.3623130322 0.8730000000

1665 3 JavaScript Cookies 0.1523251534 0.4880000000

1666 3 JavaScript Cookies 0.3568789959 0.6910000000

1667 3 JavaScript Cookies 0.1600329876 0.6980000000

1668 3 JavaScript Cookies 0.3536999226 0.6790000000

1669 3 JavaScript Cookies 0.3525381088 1.0020000000

1670 3 JavaScript Cookies 0.3506100178 0.9060000000

1671 3 JavaScript Cookies 0.3585040569 0.6880000000

1672 3 JavaScript Cookies 0.1527109146 0.5000000000

1673 3 JavaScript Cookies 0.3604559898 0.9000000000

Page 139: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CV

1674 3 JavaScript Cookies 0.3510501385 0.9320000000

1675 3 JavaScript Cookies 0.3425421715 1.0740000000

1676 3 JavaScript Cookies 0.1508738995 0.7140000000

1677 3 JavaScript Cookies 0.1489958763 0.6950000000

1678 3 JavaScript Cookies 0.3546481133 0.8480000000

1679 3 JavaScript Cookies 0.3587749004 0.8780000000

1680 3 JavaScript Cookies 0.3497700691 0.8600000000

1681 3 JavaScript Cookies 0.3545331955 0.9350000000

1682 3 JavaScript Cookies 0.3436450958 0.8600000000

1683 3 JavaScript Cookies 0.3490920067 0.6580000000

1684 3 JavaScript Cookies 0.3701169491 0.9070000000

1685 3 JavaScript Cookies 0.3568019867 0.8800000000

1686 3 JavaScript Cookies 0.3644812107 0.6720000000

1687 3 JavaScript Cookies 0.3527171612 0.8770000000

1688 3 JavaScript Cookies 0.1529090405 0.5030000000

1689 3 JavaScript Cookies 0.3629589081 0.9540000000

1690 3 JavaScript Cookies 0.3507549763 0.6530000000

1691 3 JavaScript Cookies 0.3633871078 0.8640000000

1692 3 JavaScript Cookies 0.1531500816 0.7080000000

1693 3 JavaScript Cookies 0.1517889500 0.7030000000

1694 3 JavaScript Cookies 0.3554358482 0.8710000000

1695 3 JavaScript Cookies 0.3527200222 0.8780000000

1696 3 JavaScript Cookies 0.3549501896 0.7740000000

1697 3 JavaScript Cookies 0.1521780491 0.5020000000

1698 3 JavaScript Cookies 0.3539819717 0.8590000000

1699 3 JavaScript Cookies 0.3472309113 0.8970000000

1700 3 JavaScript Cookies 0.3502559662 0.8630000000

1701 3 JavaScript Cookies 0.3529849052 0.8480000000

1702 3 JavaScript Cookies 0.3589439392 0.8990000000

1703 3 JavaScript Cookies 0.3506789207 0.8610000000

1704 3 JavaScript Cookies 0.3606810570 0.9250000000

1705 3 JavaScript Cookies 0.3730700016 0.8800000000

1706 3 JavaScript Cookies 0.3607690334 1.0090000000

1707 3 JavaScript Cookies 0.3549351692 0.8750000000

1708 3 JavaScript Cookies 0.3720910549 0.8760000000

1709 3 JavaScript Cookies 0.1527318954 0.6770000000

1710 3 JavaScript Cookies 0.1480348110 0.6890000000

1711 3 JavaScript Cookies 0.3606770039 0.9150000000

1712 3 JavaScript Cookies 0.3633258343 0.6760000000

1713 3 JavaScript Cookies 0.3466401100 0.8300000000

1714 3 JavaScript Cookies 0.3529450893 0.8990000000

1715 3 JavaScript Cookies 0.3505740166 0.9160000000

1716 3 JavaScript Cookies 0.3505179882 0.9590000000

1717 3 JavaScript Cookies 0.3522160053 0.9700000000

1718 3 JavaScript Cookies 0.3687918186 0.8740000000

1719 3 JavaScript Cookies 0.1512770653 0.6400000000

1720 3 JavaScript Cookies 0.3557188511 0.9010000000

1721 3 JavaScript Cookies 0.3543241024 0.9790000000

Page 140: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CVI

1722 3 JavaScript Cookies 0.3606431484 0.6910000000

1723 3 JavaScript Cookies 0.3543770313 0.8410000000

1724 3 JavaScript Cookies 0.3407351971 0.6760000000

1725 3 JavaScript Cookies 0.3531351089 0.6920000000

1726 3 JavaScript Cookies 0.3562850952 0.9700000000

1727 3 JavaScript Cookies 0.1586649418 0.7160000000

1728 3 JavaScript Cookies 0.3537051678 0.8780000000

1729 3 JavaScript Cookies 0.3569018841 0.8460000000

1730 3 JavaScript Cookies 0.3547959328 0.9250000000

1731 3 JavaScript Cookies 0.3568270206 0.9170000000

1732 3 JavaScript Cookies 0.1603980064 0.6700000000

1733 3 JavaScript Cookies 0.3514900208 0.7910000000

1734 3 JavaScript Cookies 0.3489880562 0.8150000000

1735 3 JavaScript Cookies 0.3545150757 0.6730000000

1736 3 JavaScript Cookies 0.1487560272 0.7990000000

1737 3 JavaScript Cookies 0.3527078629 0.8560000000

1738 3 JavaScript Cookies 0.3456091881 0.9780000000

1739 3 JavaScript Cookies 0.3567130566 0.8760000000

1740 3 JavaScript Cookies 0.1506309509 0.6520000000

1741 3 JavaScript Cookies 0.3483369350 1.0060000000

1742 3 JavaScript Cookies 0.3525099754 0.9950000000

1743 3 JavaScript Cookies 0.3550519943 0.8420000000

1744 3 JavaScript Cookies 0.3535921574 0.7100000000

1745 3 JavaScript Cookies 0.3543729782 0.7110000000

1746 3 JavaScript Cookies 0.3461430073 0.6740000000

1747 3 JavaScript Cookies 0.3527119160 0.8860000000

1748 3 JavaScript Cookies 0.3580338955 0.8940000000

1749 3 JavaScript Cookies 0.3444981575 0.8930000000

1750 3 JavaScript Cookies 0.3516721725 0.6460000000

1751 3 JavaScript Cookies 0.3534519672 0.9210000000

1752 3 JavaScript Cookies 0.3571281433 0.8600000000

1753 3 JavaScript Cookies 0.3516340256 0.9940000000

1754 3 JavaScript Cookies 0.3565161228 1.0200000000

1755 3 JavaScript Cookies 0.3449158669 0.8730000000

1756 3 JavaScript Cookies 0.3548231125 0.8840000000

1757 3 JavaScript Cookies 0.3684530258 1.0300000000

1758 3 JavaScript Cookies 0.3527898788 0.7010000000

1759 3 JavaScript Cookies 0.3527290821 0.6670000000

1760 3 JavaScript Cookies 0.3552207947 0.8670000000

1761 3 JavaScript Cookies 0.3530578613 0.8490000000

1762 3 JavaScript Cookies 0.3668909073 0.6880000000

1763 3 JavaScript Cookies 0.3425519466 0.9960000000

1764 3 JavaScript Cookies 0.3427469730 0.8940000000

1765 3 JavaScript Cookies 0.3545517921 0.9610000000

1766 3 JavaScript Cookies 0.3411810398 0.6740000000

1767 3 JavaScript Cookies 0.3560738564 0.8570000000

1768 3 JavaScript Cookies 0.3566830158 0.9680000000

1769 3 JavaScript Cookies 0.3571391106 0.8580000000

Page 141: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CVII

1770 3 JavaScript Cookies 0.3467738628 0.8620000000

1771 3 JavaScript Cookies 0.1485800743 0.6840000000

1772 3 JavaScript Cookies 0.3521420956 0.6570000000

1773 3 JavaScript Cookies 0.1547951698 0.6880000000

1774 3 JavaScript Cookies 0.1490519047 0.7740000000

1775 3 JavaScript Cookies 0.3580751419 0.8720000000

1776 3 JavaScript Cookies 0.1520080566 0.6990000000

1777 3 JavaScript Cookies 0.3590021133 0.9110000000

1778 3 JavaScript Cookies 0.3529858589 0.8850000000

1779 3 JavaScript Cookies 0.3525080681 0.8600000000

1780 3 JavaScript Cookies 0.3606839180 0.7800000000

1781 3 JavaScript Cookies 0.1505451202 0.6730000000

1782 3 JavaScript Cookies 0.3546421528 0.8420000000

1783 3 JavaScript Cookies 0.3545188904 0.8850000000

1784 3 JavaScript Cookies 0.3533349037 0.9310000000

1785 3 JavaScript Cookies 0.3569798470 0.8930000000

1786 3 JavaScript Cookies 0.3555531502 0.9500000000

1787 3 JavaScript Cookies 0.3544828892 0.8750000000

1788 3 JavaScript Cookies 0.3525509834 0.6740000000

1789 3 JavaScript Cookies 0.1530299187 0.6370000000

1790 3 JavaScript Cookies 0.3544778824 0.8790000000

1791 3 JavaScript Cookies 0.3609368801 1.0430000000

1792 3 JavaScript Cookies 0.3486270905 0.7020000000

1793 3 JavaScript Cookies 0.3547639847 0.9380000000

1794 3 JavaScript Cookies 0.3531839848 0.6610000000

1795 3 JavaScript Cookies 0.3511898518 0.7660000000

1796 3 JavaScript Cookies 0.3470809460 0.6630000000

1797 3 JavaScript Cookies 0.3592050076 0.9000000000

1798 3 JavaScript Cookies 0.3625450134 1.0460000000

1799 3 JavaScript Cookies 0.1543951035 0.6770000000

1800 3 JavaScript Cookies 0.1525478363 0.4910000000

Page 142: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CVIII

Appendix O - Search Terms for Survey

The set of search terms used for the survey.

ID Search Term

1 Php Shorthand

2 Windows Install

3 Install Linux Virtualbox

4 JavaScript Cookies

5 Apache Solr Configuration

6 Export as PDF

Page 143: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CIX

Appendix P - Survey question results

Survey question results regarding past computer experience and programming experience.

Survey Participant ID

For how many years have you

been using computers proffessionally?

(Work/School/University)

For how many years have you

been programming proffessionally?

(Work/School/University)

1 10 4

2 8 4

3 4 4

4 6 3

5 10 0

6 7 2

7 6 6

8 5 5

9 6 4

10 2 2

11 3 0

12 1 1

13 10 4

14 3 2

15 7 3

16 5 5

17 8 0

18 8 8

19 10 6

20 2 2

21 4 4

22 10 1

23 4 4

24 6 3

25 9 3

26 6 6

27 6 5

28 3 3

29 4 3

30 1 1

31 8 6

32 10 0

33 9 2

34 7 2

35 10 2

36 10 4

37 6 2

Page 144: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CX

38 7 4

39 5 3

40 7 2

41 6 5

42 8 2

43 9 5

44 3 1

45 5 0

Page 145: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CXI

Appendix Q - Survey list selection results

Survey results from the list selection.

Survey Participant ID Entry NO Query Term List Selection (CDA/original Solr)

1 1 JavaScript Cookies cda

1 2 php shorthand original Solr

2 3 JavaScript Cookies cda

2 4 Apache Solr Configuration cda

3 5 JavaScript Cookies cda

3 6 Export as PDF cda

4 7 Export as PDF cda

4 8 Install Linux Virtualbox cda

5 9 php shorthand original Solr

5 10 Apache Solr Configuration original Solr

6 11 Export as PDF original Solr

6 12 JavaScript Cookies cda

7 13 windows install cda

7 14 php shorthand original Solr

8 15 Apache Solr Configuration cda

8 16 JavaScript Cookies cda

9 17 Install Linux Virtualbox cda

9 18 Export as PDF cda

10 19 windows install cda

10 20 php shorthand original Solr

11 21 Export as PDF cda

11 22 Install Linux Virtualbox original Solr

12 23 Apache Solr Configuration original Solr

12 24 Export as PDF cda

13 25 JavaScript Cookies original Solr

13 26 Install Linux Virtualbox original Solr

14 27 Install Linux Virtualbox cda

14 28 JavaScript Cookies original Solr

15 29 php shorthand cda

15 30 Install Linux Virtualbox cda

16 31 Apache Solr Configuration cda

16 32 php shorthand cda

17 33 windows install cda

17 34 Apache Solr Configuration original Solr

18 35 Export as PDF cda

18 36 windows install cda

19 37 Apache Solr Configuration cda

19 38 windows install cda

20 39 Apache Solr Configuration cda

Page 146: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CXII

20 40 Export as PDF cda

21 41 windows install cda

21 42 Export as PDF cda

22 43 Export as PDF cda

22 44 Install Linux Virtualbox cda

23 45 Apache Solr Configuration original Solr

23 46 windows install original Solr

24 47 JavaScript Cookies cda

24 48 Apache Solr Configuration original Solr

25 49 php shorthand original Solr

25 50 windows install cda

26 51 php shorthand original Solr

26 52 JavaScript Cookies cda

27 53 JavaScript Cookies original Solr

27 54 Apache Solr Configuration cda

28 55 Install Linux Virtualbox original Solr

28 56 php shorthand cda

29 57 php shorthand original Solr

29 58 Export as PDF cda

30 59 Export as PDF cda

30 60 JavaScript Cookies cda

31 61 Install Linux Virtualbox cda

31 62 Apache Solr Configuration original Solr

32 63 windows install original Solr

32 64 Install Linux Virtualbox cda

33 65 Install Linux Virtualbox cda

33 66 JavaScript Cookies cda

34 67 Install Linux Virtualbox cda

34 68 Export as PDF cda

35 69 Install Linux Virtualbox cda

35 70 php shorthand cda

36 71 Install Linux Virtualbox cda

36 72 windows install original Solr

37 73 Export as PDF cda

37 74 php shorthand cda

38 75 Apache Solr Configuration cda

38 76 Install Linux Virtualbox cda

39 77 Export as PDF cda

39 78 php shorthand cda

40 79 windows install original Solr

40 80 php shorthand cda

41 81 windows install cda

41 82 Apache Solr Configuration cda

42 83 php shorthand original Solr

Page 147: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CXIII

42 84 JavaScript Cookies original Solr

43 85 JavaScript Cookies cda

43 86 windows install original Solr

44 87 Apache Solr Configuration original Solr

44 88 windows install cda

45 89 JavaScript Cookies cda

45 90 windows install original Solr

Page 148: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CXIV

Appendix R - Recorded clicks

Contains all of the clicks that was recorded and used during all measurements.

ID Document ID Clicks Full Query Term Term

1 2604727 1 windows install windows

2 2604727 1 windows install install

3 77005 3 windows install windows

4 77005 3 windows install install

5 1967370 1 windows install windows

6 1967370 1 windows install install

7 30441122 1 tomato tomato

8 1722726 1 signature signature

9 1255472 1 mushroom soup mushroom

10 1255472 1 mushroom soup soup

11 5311334 1 export pdf export

12 5311334 1 export pdf pdf

13 12203898 1 tomato soup tomato

14 12203898 1 tomato soup soup

15 13581358 1 virtualbox windows virtualbox

16 13581358 1 virtualbox windows windows

17 39480185 1 php myadmin php

18 39480185 1 php myadmin myadmin

19 16323484 11 windows install windows

20 16323484 11 windows install install

21 889348 5 windows install windows

22 889348 5 windows install install

23 40384975 1 windows install windows

24 40384975 1 windows install install

25 39557844 2 windows install windows

26 39557844 2 windows install 2 install

27 39557844 2 windows install 2 2

28 32743001 3 windows install 2 windows

29 32743001 3 windows install 2 install

30 32743001 2 windows install 2 2

31 20697824 7 windows install windows

32 20697824 7 windows install install

36 11445037 3 windows install windows

37 11445037 3 windows install install

38 8876051 2 install install

39 6277222 1 use php shorthand use

40 6277222 1 use php shorthand php

Page 149: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CXV

41 6277222 1 use php shorthand shorthand

42 8147343 1 use php shorthand use

43 8147343 2 use php shorthand php

44 8147343 2 use php shorthand shorthand

45 5208015 1 use php shorthand use

46 5208015 1 use php shorthand php

47 5208015 1 use php shorthand shorthand

48 19318271 1 install windows phone install

49 19318271 1 install windows phone windows

50 19318271 1 install windows phone phone

51 6412083 1 use php shorthand use

52 6412083 1 use php shorthand php

53 6412083 1 use php shorthand shorthand

54 7727607 1 use php shorthand use

55 7727607 1 use php shorthand php

56 7727607 1 use php shorthand shorthand

57 23776290 1 windows install windows

58 23776290 1 windows install install

62 17268210 1 windows install windows

63 17268210 1 windows install install

64 13430590 1 mac os install mac

65 13430590 1 mac os install os

67 10880675 51 windows install windows

68 10880675 51 windows install install

69 14286681 42 windows install windows

70 14286681 42 windows install install

71 28981078 26 windows install windows

72 28981078 26 windows install install

73 16826822 20 php shorthand php

74 16826822 20 php shorthand shorthand

75 1068750 1 php shorthand php

76 1068750 1 php shorthand shorthand

77 24286340 67 php shorthand php

78 24286340 67 php shorthand shorthand

79 14038322 31 php shorthand php

80 14038322 31 php shorthand shorthand

81 27736900 1 php shorthand php

82 27736900 1 php shorthand shorthand

83 19790530 24 php shorthand php

84 19790530 24 php shorthand shorthand

85 6442315 40 javascript cookies javascript

86 6442315 40 javascript cookies cookies

87 12819594 11 javascript cookies javascript

Page 150: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CXVI

88 12819594 11 javascript cookies cookies

89 16669478 36 javascript cookies javascript

90 16669478 36 javascript cookies cookies

91 6448427 30 apache solr configuration apache

92 6448427 30 apache solr configuration solr

93 6448427 30 apache solr configuration configuration

94 1559528 1 apache solr configuration apache

95 1559528 1 apache solr configuration solr

96 1559528 1 apache solr configuration configuration

97 7157954 18 apache solr configuration apache

98 7157954 18 apache solr configuration solr

99 7157954 18 apache solr configuration configuration

100 21324769 22 apache solr configuration apache

101 21324769 22 apache solr configuration solr

102 21324769 22 apache solr configuration configuration

103 12822152 5 apache solr configuration apache

104 12822152 5 apache solr configuration solr

105 12822152 5 apache solr configuration configuration

106 10101462 40 install linux virtualbox install

107 10101462 40 install linux virtualbox linux

108 10101462 40 install linux virtualbox virtualbox

109 13291015 10 install linux virtualbox install

110 13291015 10 install linux virtualbox linux

111 13291015 10 install linux virtualbox virtualbox

112 37815644 15 install linux virtualbox install

113 37815644 15 install linux virtualbox linux

114 37815644 15 install linux virtualbox virtualbox

115 35007852 31 install linux virtualbox install

116 35007852 31 install linux virtualbox linux

117 35007852 31 install linux virtualbox virtualbox

118 17094800 20 export as pdf export

119 17094800 20 export as pdf as

120 17094800 20 export as pdf pdf

121 21629877 5 export as pdf export

122 21629877 5 export as pdf as

123 21629877 5 export as pdf pdf

124 13428473 29 export as pdf export

125 13428473 29 export as pdf as

126 13428473 29 export as pdf pdf

127 4487427 1 export as pdf export

128 4487427 1 export as pdf as

129 4487427 1 export as pdf pdf

130 19915811 1 export as pdf export

Page 151: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CXVII

131 19915811 1 export as pdf as

132 19915811 1 export as pdf pdf

133 59436 1 java get time java

134 59436 1 java get time get

135 59436 1 java get time time

Page 152: Using Clickthrough Data to optimize search result rankinghis.diva-portal.org/smash/get/diva2:1139938/FULLTEXT01.pdf · 2.1.2 Apache Solr Apache Solr is an open source enterprise search

CXVIII

Appendix S - Server Specifications

The hardware specifications of the server that was used for the study.

CPU Intel Core i7-4790K Devlis Canyon Edition @ 4.2GHz

RAM 16GB (2x8GB) Corsair CL10 1833MHz VENGEANCE LP

Motherboard Gigabyte GA-Z97X-Gaming 7

Hard drive Samsung SSD 840 EVO 250GB

Network Speed (Up-Down) 100Mbit/s - 100Mbit/s

Operating System Ubuntu 16.04 LTU

Web Server Apache2