managing your own data collection with open data...

Managing your own data collection with Open Data Kit

Marco Minghini – GEOlab, Politecnico di Milano | Como Campus

Como – July 12, 2015

COST Action TD1202 –Mapping and the Citizen Sensor

Training School “FOSS4VGI – Using Free and Open Source Software with VGI: integration, analysis and visualization”

Open Data Kit✔ Open Data Kit (ODK) is a suite of tools allowing to plan, execute

and manage data (even VGI) collection campaign:

✗ developed by Washington University – Department of Computer Science & Engineering, which leads the project together with Change (http://change.washington.edu)

✗ born in 2008 as a google.org sponsored project, currently funded by a Google Focused Research Award as well as user donations

✗ used in thousands of projects all around the world (see http://opendatakit.org/about/deployments)

✗ can count on a large community of users and developers✗ open source under the Apache2 license

(http://www.apache.org/licenses/LICENSE-2.0)✗ based on open standards (e.g. Xforms from W3C)

http://opendatakit.org

http://change.washington.edu/

http://opendatakit.org/about/deployments

http://www.apache.org/licenses/LICENSE-2.0

http://opendatakit.org/

Open Data Kit✔ Open Data Kit suite is composed of 3 main modules allowing to:

✗ create the form for data collection

http://opendatakit.org

ODK Build: HTML5 Web interface (for simple forms) XLSForm: converter of forms created in Excel (for complex forms)

✗ compile the form on the field ODK Collect: Android app

✗ store, visualize and export collected data ODK Aggregate: server platform

COLLECT

FORM COMPILATION

AGGREGATE

FORM STORAGE

BUILD

FORM CREATION


ODK Build✔ ODK Build is a Web application featuring a drag-and-drop user

interface which is recommended to create simple forms:

✗ HTML5 application, it uses Ruby and JavaScript

✔ It can be used in 2 ways:

✗ download the source code and run it locally (offline use)✗ open the Web page http://build.opendatakit.org and create/use

an account (online use) → SUGGESTED!

http://opendatakit.org/use/build

http://build.opendatakit.org/


ODK Build http://opendatakit.org/use/build

✔ To add an element (question) to the form:

✗ click on the element of interest (for instance Text) in the bottom part of the screen



✔ To add an element (question) to the form:

✗ define its properties on the right side of the screen

name of the field inthe exported XML file

name of the field aspresented to the user

suggestion given tothe user for the field

mandatory answerfor the field

default value forthe field

min/max length anderror message

read-only optionfor the field



✔ Add a question on the age (Numeric field):

✗ define the properties of the field on the right side of the screen










✔ Add a question on the date of birth (Date field):











✔ Add a question on the place of residence (Location field):










✔ Ask for the upload of a picture (Media field):







type of multimedia:picture, audio, video




✔ Add a question on the native language (Choose One field):







list of possibleanswers




✔ Add a question on the languages spoken (Select Multiple field):







list of possibleanswers




✔ Edit and save the form you have created:

✗ change the order of the questions by dragging the elements in the desired positions

✗ choose a name for the form by clicking the rename button on the top left, then enter the desired name and press done

✗ save the form from the menu File → Save (or Save Form As). The form will remain stored on your ODK Build account.

✗ export the form in the XForms format from the menu File → Export to XML, then select Download in the following window (from where you can visualize the form XML code)

→


XLSForm http://opendatakit.org/use/xlsform

✔ XLSForm is a tool which converts forms created from an .xls file (e.g. in Excel) in the XML Xforms standard. It is recommended for the creation of more complex forms:

✗ to be read from the XLSForm converter, the .xls file must have a proper structure described at the page http://xlsform.org

✗ some useful examples (in addition to the one described in the following) on how to create the .xls file are available at the same page http://xlsform.org

✔ In the following we will first create the same form which was previously created using ODK Build. Then we will explore some additional functionalities which are not supported from ODK Build.

http://opendatakit.org/use/xlsform

http://xlsform.org/

http://xlsform.org/


✔ Open Excel or LibreOffice Calc and save the file with an .xls extension. We are then going to work on this file.

✔ Create 3 worksheets and name them as follows:

✗ survey: contains the general structure of the form, including the list of questions and the related options

✗ choices: contains the list of the possible answers for multiple choice questions

✗ settings (optional): contains additional options for the form



✔ In the survey worksheet each row corresponds to a question

✔ This worksheet has 3 required columns (i.e. each row must have a value in each of these columns):

✗ type: specifies the type of question (text, number, date, etc.)✗ name: specifies the unique identifier of the question✗ label: specifies the text of the question as presented to the user

✔ The columns of the worksheet can appear in any order.

✔ The worksheet formatting (cell dimension, background color, font type/color/dimension, etc.) will be ignored by XLSForm and thus can be customized:



✔ Add a question asking for the name (type text)

✔ Add a question asking for the age (type integer)

✔ Add a question asking for the date of birth (type date)

✔ Add a question asking for the place of residence (type geopoint)



✔ Add a question asking for a picture (type image)

✔ Add a question asking for the native language (type select_one)

✔ Add a question asking for the languages spoken (type select_multiple)



✔ In the choices worksheet each row corresponds to a possible answer for multiple choices questions (both of type select_one and select_multiple)

✔ The choices worksheet has 3 required columns (i.e. each row must have a value in each of these columns):

✗ list name: points to the question the possible answers refer to✗ name: specifies the unique identifier of the answer✗ label: specifies the text of the question as presented to the user

✔ The columns of the worksheet can appear in any order.

✔ The worksheet formatting (cell dimension, background color, font type/color/dimension, etc.) will be ignored by XLSForm and thus can be customized:



✔ Add the possible answers for the question asking for the native language (whose column text of the survey worksheet had the identifying name nat_lan)

✔ Add the possible answers for the question on the languages spoken (whose column text of the survey worksheet had the identifying name nat_spoken)



✔ Add the hint that will be presented to the user together with each question:✗ in the survey worksheet, create a column named hint (optional)

which contains the hints

✔ Make the answer to one or more questions mandatory:✗ in the survey worksheet, create a column named required

(optional) and write yes for the desired questions



✔ Make the form available in two languages (English and your own native language):✗ in the survey worksheet

rename the label column into label::English create a new column named label::XXX (where XXX is your native

language) and fill it with the texts of the questions as presented to the user

rename the column hint into hint::English create a new column named hint::XXX, and fill it with the texts

of the hints for the questions as presented to the user



✔ Make the form available in two languages (English and your own native language):✗ in the choices worksheet

rename the label column into label::English create a new column named label::XXX (where XXX is your native

language) and fill it with the texts of the answers as presented to the user



✔ Make the form available in two languages (English and your own native language):✗ in the settings worksheet, create 3 columns

form_title: form name as presented to the user form_id: form name in the XML file default_language: default language of the form



✔ Add a question based on the specific answer given by the user to a previous question (for instance on the native language):✗ in the survey worksheet

in the desired position, add the row corresponding to the new question

create a new column, named relevant, which tells the question (identified by the name column of the survey worksheet) and its answer (identified by the value of the name column of the choices worksheet) to which the new question is conditioned, using the proper syntax shown in the figure below



✔ Add icons that users will visualize next to the possible answers to multiple choice questions (for instance, for native language equal to Italian and for native language different than Italian):✗ the images (italia.png and mondo.png) can be downloaded from

http://geomobile.como.polimi.it/FOSS4VGI/ODK_data.zip✗ in the choices worksheet

add two columns named image::English and image::XXX (where XXX is your native language) and write the names (with extension) of the images, in correspondence to the answers of interest



✔ Convert the .xls file in the standard format XML XForms:

✗ open the page http://opendatakit.org/xiframe/✗ hit the Browse and select the .xls file you have created✗ hit the Submit button to submit the .xls file: possible mistakes in

the .xls file will be found at this point✗ hit the Download button to download the form in XML format,

and specify the file name and path✗ hit the Preview in enketo button to visualize a preview of form

compilation using Enketo (a Web framework for managing forms)



✔ By default, the Android ODK Collect app will use – as methods to determine the position – the mobile device geolocation services (GPS, cellular network and Wi-Fi).

✔ to set the position through an interactive map on the mobile device, you need to directly edit the XML file:

✗ open the XML file with a text editor✗ find the row of the XML code corresponding to the compilation

of the place of residence field (which is identified by the value of the name column in the survey worksheet for the question on the place of residence)

✗ within the <input> tag, add the string appearance= "placement-map" as shown below

✗ save the modified XML file


ODK Aggregate✔ ODK Aggregate is a server platform written in Java, allowing to:

✗ provide blank forms to, and get finalized forms from ODK Collect (or any other client based on the XForms standard)

✗ store the data collected (in synchronization with a database)✗ visualize the data collected (also through graphs)✗ export the data collected (in the CSV, KML and JSON formats)✗ publish the data collected in other formats (Google Spreadsheets

and Google FusionTables)✗ manage users and the related permissions

http://opendatakit.org/use/aggregate


ODK Aggregate✔ Can be installed and configured in 3 ways:

✗ on a cloud platform (Google App Engine, Amazon EC2, etc.)✗ on a local Tomcat server (only v. 6 is currently supported) and

synchronized to a MySQL or a PostgreSQL database✗ on a virtual machine (already configured, works even offline),

downloadable from https://gumroad.com/l/odk-aggregate-vm


✔ In this practice, we will use the option 2:

✗ ODK Aggregate running under Tomcat and synchronized to a PostgreSQL database

✗ accessible at http://131.175.143.49/ODKAggregate click on Log In on the top right corner hit the button Sign in with Aggregate password use the following credentials: username: foss4vgi, password: cost

https://gumroad.com/l/odk-aggregate-vm


http://131.175.143.49/ODKAggregate

ODK Aggregate✔ The Web interface is composed of 3 sections:


✗ Form Management: allows to visualize the available forms and modify their properties; add new forms; publish, export and delete forms;

✗ Submissions: allows to visualize, for each available form, the contents of user submissions; publish and export forms


ODK Aggregate✔ The Web interface is composed of 3 sections:


✗ Site Admin (accessible for users with administrator permissions): allows to create new users (with their permissions) of type Data Collector, able to download blank forms and send finalized

forms from ODK Collect Data Viewer, able to also filter, visualize and export the results of

the submissions from ODK Aggregate Form Manager, able to also upload new forms, edit their options,

delete blank forms and user submissions Site Administrator, able to also manage users and permissions


ODK Aggregate http://opendatakit.org/use/aggregate

✔ To upload the form previously created:

✗ in the Form Management section, hit the Add New Form button in the top left corner

✗ on the following window in correspondence of Form definition (xml file):, hit the Browse

button and select the previously created XML file in correspondence of Optional Media file(s):, hit the Browse

button and select the images italy.png, world.png and form_ logo.png, which will be used as form logo (NOTE: these must be the name and extension of the image used as logo!)

hit the Upload Form button and wait for the upload



✔ To visualize the user submissions:

✗ in the Submissions section and Filter Submissions subsection

select the form of interest from the Form menu on the top left on the main window the contents of user submissions are shown the button allows to delete the single submissions (provided

that you have the permissions required to do that)



✔ To export user submissions:


hit the Export button on the top right select the type of file (CSV, KML, JSON) and press Export

✗ exported submissions will be listed in the Submissions section and Exported Submissions subsection



✔ To publish user submissions:


hit the Publish button on the top right select the desired type of publication (Google FusionTables,

Google Spreadsheet, etc.) and press Publish



✔ To visualize user submissions in graphs:


hit the Visualize button on the top right select the type of graph (Pie Chart or Bar Graph) in the Type menu

and the attribute of interest in the Column to Visualize menu


ODK Collect http://opendatakit.org/use/collect

✔ ODK Collect is an Android app which communicates with ODK Aggregate for the download of blank forms and the upload of finalized forms:

✗ client compatible with the XForms standard✗ interprets the forms as a sequence of prompts✗ allows to save the form at each step✗ form compilation is also possible offline

✔ Download ODK Collect on an Android device:

✗ from the official Google Play store https://play.google.com/store/apps/details?id=org.odk.collect.android

✗ as an Android application package (.apk) from http://opendatakit.org/downloads/download-info/odk-collect-apk

http://opendatakit.org/use/collect

https://play.google.com/store/apps/details?id=org.odk.collect.android

http://opendatakit.org/downloads/download-info/odk-collect-apk


✔ To connect ODK Collect to the ODK Aggregate server:

✗ hit the Options device button, then select General Settings✗ select Configure platform settings✗ insert the following parameters

URL: http://131.175.143.49/ODKAggregate Username: foss4vgi Password: cost


http://131.175.143.49/ODKAggregate


✔ To download the form from ODK Aggregate and compile it:

✗ from the main page select Get Blank Form✗ select the form of interest and press Get Selected✗ from the main page select Fill Blank Form✗ on the following page select the form you wish to compile



✔ To compile the form:

✗ answer the questions (at least the required ones, otherwise you can't go on) providing the required inputs (text, integer number, date, picture, position, choice of one or more options from a list)



✔ To change the language of the form:

✗ in any moment during form compilation, hit the Options device button, select Change Language and choose the language

✗ the form will be now visualized in the desired language



✔ To save the form and send it to the ODK Aggregate server:

✗ at the end of form compilation, give a name to the submission and hit the Save Form and Exit button

✗ from the main page select Send Finalized Form (1)✗ Select the form you wish to send and press Send Selected


✔ Collected data is stored on the PostgreSQL database to which ODK Aggregate is synchronized (and on which it writes):

✗ during ODK Aggregate configuration, a predefined set of tables is created in the database to store general information (server preferences, security options, user credentials, etc.)

✗ for each form created in ODK Aggregate, the database will have:


one main table to store all the contents except multimedia 3 additional tables for each multimedia (picture, video or audio)

present in the form


✔ On your Android device, download the form COST_official from ODK Aggregate. This form is a simplified version of the previous one, which asks for (answers to all questions are required):

✗ gender (male or female) – type select_one✗ age – type integer✗ city of residence – type text✗ place of residence – type geopoint✗ native language (English, Spanish, French, German, Italian, other)

– type select_multiple

Outdoor exercise

✔ Interview 10 people while walking around Como city, and send the submissions to the ODK Aggregate server.

Outdoor exercise✔ All the data you are going to collect will be visible on a Web map

at http://geomobile.como.polimi.it/FOSS4VGI/ODK_map.html

✔ The map results from the open source architecture shown below

SERVER CLIENT

template form

GeoJSON

AGGREGATE COLLECTcompiled form

http://geomobile.como.polimi.it/FOSS4VGI/ODK_map.html

References✔ ODK official website:

✗ http://opendatakit.org

✔ W3C (World Wide Web Consortium) XForms Standard v. 1.1:✗ http://www.w3.org/TR/xforms

http://creativecommons.org/licenses/by-nc-sa/3.0

Marco Minghini, PhDGEOlab – Politecnico di Milano, Como Campus

[email protected]@MarcoMinghini


http://www.w3.org/TR/xforms

http://creativecommons.org/licenses/by-nc-sa/3.0

mailto:[email protected]

managing your own data collection with open data...

Documents