submitted as milestone 2.10 under anlec project 0128 sub...

FINALREPORT

OCTOBER2015

SubmittedasMilestone2.10under

ANLECProject0128

Sub-Project2

Preparedby

DrAdrianSheppard

DrDanielPalamara

DrOlafDelgado-Freidrichs

“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”

©ANU2015|Page2


©ANU2015|Page3

TableofContents

1.Background 4

1.1ProjectHistory 5

2.Outputs 6

2.1TheDDCA 6

2.2TheData 7

2.3TheUserGuide 11

2.4AUserWorkshop 11

2.5FutureWork 11

3.AppendixA.UserGuide 12


©ANU2015|Page4

1. Background

ThisdocumentreportsonMilestone2.10ofANLECProject128:Maximisingthevalueof

digitalcoreanalysisforcarbonsequestrationsiteassessment.Thedocumentmilestone

objectivesare:

2.10Fullinternetaccessdigitaldatabasewithallconventionalanddigitalcoreanalysisdata,

userguide,andworkshoponsoftwareforallusers.

TheDigitaldatabaseisamajorcomponentofsub-project2andhaslinkstotheoutputof

othersub-projectsinANLECProject128:

Sub-Project1.Toderiveafullsuiteofspecialcoreanalysis(capillarypressure,supercritical

CO2:brinerelativepermeability)datasetsonplugsfromSuratbasinreservoirandsealrock

samples;

Sub-Project2.Toobtainhigh-resolution3Ddigitalimagesofthesameplugsamples,derive

digitalpetrophysicalandSCALdata,andperformpetrographicanalysesbyautomated

quantitativemineralmapping.Also,buildadigitaldatabaseofconventionalandDCA-

deriveddata,rocktypes,3Dimages,andstorageandsealpropertiesofcorematerialfrom

theSuratbasin;

Sub-Project3.Perform3DimagingofinsitusupercriticalCO2saturationattheporescale.

Correlateobservedsaturationrelationshipswithporositydistributionsandrocktypes;

Sub-Project4.Derivefasterandcheapermodel–basedanalysesofflow,storageand

CO2:brinedisplacementsindifferentrocktypesviareliableimage–basedmodelling;

Sub-Project5.Undertaketimeseries(4D)imagingandconventionalexperimentalstudiesto

measurethegeochemicalreactivityanddissolutiontrappingcapacityofcorematerialusing

supercriticalCO2;and

Sub-Project6.Integrateinformationobtainedatlaboratoryscaletowirelinelogdata.


©ANU2015|Page5

Considerableachievementshavebeenmadeonthedigitaldatabase(hereafterreferredto

astheDDCA–DatabaseforDigitalCoreAnalysis);thisdocumentliststheoutputsofthis

projectwithinthecontextoftheMilestonedeliverables.

1.1 Project History

Amilestonereportfordeliverable2.4,entitled“Finalspecificationdocumentwithan

overviewoffunctionalityandarchitecturefordigitaldatabase”,wassubmittedtoANLECin

June2013.

Additionally,twoeducationsessionswereconducted.Afirsteducationsessionwas

providedforANLECstaffon6May2014.Thismeetinginvolvedalivedemonstrationofthe

DDCA,specificallybrowsingandenteringdata.Anoverviewofthestructurewaspresented

aswellasaschemaforhowthesystemwouldbeimplemented,whichincludeddatafrom

theSuratBasin.Theattendees,SandeepSharma,programmanagerfortheSouth-WestHub

projectandRickCausebrookandMelodyXiuhuiLifromANLECR&Dprovidedfeedbackon

variousaspectsofthedatabaseimplementation.

Asecondeducationsessionwasprovidedtoalargergroupon5June2014aspartofthe

2014ANLECR&DEastCoastReview.TheaudienceincludedANLECR&Dstaff(Noel

Simiento,RickCausebrookandMelodyLi),CTSCoTechnicalProgramManagerRobHeath,

formerANLECR&DResearchManagerJimUndershultz,aswellasindividualsfromother

relatedANLECprojects.ThescopeandpurposeoftheDDCAwasdiscussedatthismeeting,

particularlyhowitwouldbeimplemented,howstakeholderswouldaccessdata,whatits

currentandplannedcapabilitiesare(includingitssearchcapabilities)andwhatiswithinthe

scopeoftheDDCAproject.

AfurtherreportwassubmittedtoANLECfordeliverable2.7,entitled“DraftUserGuide”in

June2014.Thatreportoutlinedtheimplementationofthemajorityofthecomponents

describedinthespecificationdocumentfrom2013andalsodescribedthedatastructure

andvisualisationworkflowfortheimagedatathataccompaniesthemetadatainthe

database.Itprovidedacomprehensiveoverviewoftheconceptualandtechnicalframework

ofthedatabase.Assuch,thosedetailshavenotbeenrepeatedinthismilestonereport.


©ANU2015|Page6

2. Outputs

2.1 The DDCA

Thefirstandmostimportantoutputisthedatabaseitself.ThisiscurrentlyhostedatNCI,

theNationalComputationalInfrastructure,andisaccessedwiththefollowingaddress:

https://ddca-staging.anu.edu.au

Thereisnolandingpageprovided,soonlyauthorisedusersareabletoaccessanypartof

thedatabase.AsmentionedtoANLECinface-to-facediscussions,thereisscopeforANLEC

orotherpartiestodeploytheirownversionofthedatabaseandhostthedataontheirown

server.ThiswouldmakeitmoredifficulttoprovidetheongoingsupportthattheANUand

NCIarepreparedtoprovideinordertokeeptheDDCAsiterunningforsomeyears.

AnotheroptionisforthesitetocontinuetobehostedatNCI,butaccessedthroughan

anlecrd.com.auwebaddress.

Asoutlinedinpreviousdocumentsandproposals,thedatabasefeatures:

(1) Thecapabilitytoautomaticallyharvesttomogramdataandotherproductsrelatedto

thedigitalcoreanalysisprocess.Thisisapowerfulanduniquecomponentofthe

database,andanareawheremanyfuturerefinementsarepossibleinordertomore

closelycoupletheuser-enteredmetadatawiththeautomatically-harvestedimage

data.

(2) AuthenticationprovidedviaGoogle'sgmailauthenticationprocess,whichmandates

thatallusershaveaGmailaccountforaccess.Asof2015thisappearstobethemost

secureexternalauthenticationsystemavailable.Asmentionedinthemilestone2.4

report,itwouldbeimpracticalandriskytodevelopaninternalauthentication

systemfortheDDCA.Accesstoindividualelementsisdeterminedbygroup

ownershipofeachelementandwhethertheuserisamemberofthatgroup.

(3) Comprehensivesearchandexportfunctionality,allowingusersandanalyststo

searchacrossalldataelementtypeswithinaprojectandexportdatawithuser-

selectedattributes.Thereisalsoscopeforthedata,exportedinCSV(orinJSON,that

is–JavaScriptObjectNotation)tobeamendedoutsideoftheDDCAandreimported.

Customsearchescanalsobedefinedandsavedforfutureuse.


©ANU2015|Page7

(4) Intuitiveandflexiblebrowsingcapabilities,includingtheabilitytoquicklynavigateto

parentorchildelementsaswellassiblingelements,ortorapidlynavigateeven

furtherthroughthechain,toancestorordescendent.

(5) Qualitycontrolandcurationfunctionality,includingtheabilitytomodifyanddelete

elements,attachcustomdata,andmoveelementswithinthedatahierarchy.All

operationsarereversible,withnodataeverbeingpermanentlydeletedfromthe

DDCA.

Assuch,theDDCAnowsatisfiesallofthekeyfeaturesindicatedunderthe“Summaryof

Requirements”frompoint1.2inANLECMilestoneReport2.4(page4),whicharelisted

below:

ü Theabilitytostoreallrelevantdatatypesandtheirinter-relationships

ü Asecuritymodelthatincludesmixed-permissions

ü Datauploadcapabilities(automaticandmanual)

ü Extensibility(suchastheabilitytoincludecustomdata)

ü Dataqualitycontrolandcurationcapabilities

ü BrowsingandSearching

ü Inspectingandanalysingdataelements

2.2 The Data

ThenextcriticalcomponentoftheDDCAdeliverableisthecontent,whichcentresonthe

conventionalanddigitalcoreanalysisdatafromthevarioussub-projectsofANLEC0128,as

showninFigure1.Thesediversedatahavebeenuploadedintothedatabaseinvarious

forms,andaccompanythelargestoreoftomogramandotherimagedatathatare

generatedbythedigitalcoreanalysisworkflowthathasbeenappliedtothevarious

samples.

Mostofthedataofinterestaredirectlyattachedtotherelevantsub-plugs(informationon

searchingandnavigatingtosub-plugsisincludedintheDataGuide)asshowninFigure2,

andincludeimagesofthecoreplugsandscopingscansaswellasX-andZ-slicesofthesub-


©ANU2015|Page8

plugsshowingmicroporosity,grainsizedistributions,mineralsegmentation,saturatedand

nativestatetomograms,porosityprofilesandmore.


©ANU2015|Page9

Figure1.AnalysesperformedunderANLEC0128

3 4 6

SEM

+ QE

MSCA

N R

CA SC

AL W

hole-

Core μC

T Sc

oping

Scan

μCT

Pore

-scale

μCT

+ Seg

. W

hole-

Core μC

T Sc

oping

Scan

μCT

QEM

SCAN

+ Ge

o.

SEM

+ QE

MSCA

N R

CA SC

AL W

hole-

Core μC

T Sc

oping

Scan

μCT

Pore

-scale

μCT

+ Seg

. C

ore-flo

oding

Dyna

mic M

odelli

ng W

hole-

Core μC

T Sc

oping

Scan

μCT

QEM

SCAN

+ Ge

o. U

pscali

ng

H WC15 800.8 P2 1168.0P3 1170.0

E1 946.7 P5 1174.0E2 954.4 P6 1176.0 E3 961.9 P7 1178.2 E4 987.3 P8 1180.4 E5 988.7 P9 1182.0E8 1002.6 P10 1184.0 E9 1006.0 P11 1186.0 E10 P12 1188.0 E11 1032.1 P13 1190.2 E12 1035.0 P14 1192.0 E13 1038.0 P15 1194.0 E14 1039.9 P17 1200.0 E15 1040.7 P19 1204.0 E16 1043.7 P20 1206.0 E17 1048.7 P21 1207.9E18 1056.2 P22 1210.0 E19 1152.5 P23 1212.0E20 1155.4 P24 1214.0 E22 1161.8 P25 1219.0E23 954.0 P26 1220.9E24 966.0 P28 1224.9 E25 997.0 P29 1227.0 E26 1015.0 P30 1229.1 E27 1026.0 P31 1231.0 E28 1030.0 P32 1233.1 E29 1151.0 P33 1235.0 E30 1152.0 PU1 1163.2E32 1161.0 PU2 1163.9 E33 1163.0 PU3 1164.3 WC3 981.2 PU4 1164.6WC5 992.4 PU5 1174.4 WC8 1043.7 PU6 1174.5 WC9 1056.1 PU7 1195.4

PU8 1195.5 Notes: H = Hutton Sandstone PU9 1216.1

Seg. = Segmentation PU10 1216.3 Geo. = Geochemical PU11 1216.6

PU12 1216.8 PU13 1217.1 PU14 1217.3 PU15 1217.6 PU16 1217.8 WC11 1165.4WC14 1217.5

Sub-Project

Precip

ice Sa

ndsto

neEvergr

een F

ormati

on

Sub-Project

IDDepth(m)

1 2 5

IDDepth(m)

1 2 5


©ANU2015|Page10

Figure2.SnapshotoftheDDCAshowingsomeofthedatatypicallyenteredforasub-plug(PrecipicesampleP22).Notethepresenceofcalculateddataenteredascustomdata(aplotofformationfactor)aswellasdescendantdatarelatingtoimageslicesandmetadatafromtheautomaticallyingestedtomograms.


©ANU2015|Page11

2.3 The User Guide

Overall,theDDCAinterfaceisintuitiveand–becauseofthehierarchicalnatureofthedata–

straightforwardtonavigate.

Tofacilitatefamiliarisationwiththewebsite,auserguidehasalsobeenprepared–itis

presentedinAppendixA.Itfeaturesaquick-startguidetoallowuserstoquicklyaccessthe

searchandbrowsingfunctionalityoftheDDCA.

ItisintendedthattheUserGuidewillevolveandadaptwithincreasingclientuseofthe

DDCA.Itwill,overtimeandwithuserfeedback,bereplacedwithcontext-sensitiveinline

helpbuiltdirectlyintotheDigitalDatabaseforCoreAnalysis.Inthemeantime,itspurposeis

toprovidedirection(andinstructionsforeasyaccess)onthemainfeaturesoftheDDCA–

browsingandsearchingmetadatarelatedtodigitalcoreanalyses.Foramoredetailed

understandingofthephilosophyofthedatabaseanditstechnicalframework,usersshould

refertotheMilestone2.7report.

2.4 A User Workshop

WiththeDDCAfullyfunctionalandconsiderableamountsofdatauploaded,thedatabaseis

readytobedemonstratedinauserworkshop.Onewebinar,presentedbyAdrianSheppard,

wasorganisedbyANLECR&Dheldon6thNovember2016.Furtheruserconsultationwill

yieldimportantinsightsintouserexpectationsandareasinwhichthedatabasecanbemade

moreuser-friendlyandmorefunctional.AwebcastoftheDDCAinoperationwillbe

producedinearly2016.

2.5 Future Work

TheDDCAtodayrepresentsasecuredatabasethatisabletostoreandpresentdiversedata.

Itisbuiltonasophisticatedback-endarchitecture.Itwillenablethedatageneratedinthis

largeprojecttodelivervaluetotheend-usersoftheSuratbasinflagshipproject.Withthe

essentialframeworkinplace,theDDCAisnowatapointwherearelativelysmalladditional

effortcouldyieldmajorimprovementsindatapresentationandtheuser-friendliness,

pendingsufficientuserfeedback.Itisthereforerecommendedthat,ifpossible,some

additionalresourcesbeallocatedtotheongoingdevelopmentofthedatabaseinresponse

touserfeedback.


©ANU2015|Page12

3. Appendix A. User Guide

Seefollowingpages

DatabaseforDigitalCoreAnalysis

(DDCA)USERGUIDE

VERS I ON 1 . 0

It is intended that this User Guide will evolve and adapt with increasing client use of the DDCA. It will, over time, be replaced with context-sensitive inline help built directly into the Digital Database for Core Analysis. In the meantime, its purpose is to provide direction and instructions for easy access on the main features of the DDCA – browsing and searching metadata related to digital core analyses.

January 2016

DDCA U s e r G u i d e 0 . 9

User Guide / Page 2

QuickStartGuideThe DDCA (Database for Digital Core Analysis) stores metadata from the digital and physical core

analysis workflows. Each entity in the hierarchy is referred to as a data element and represents a

conceptual and sometimes physical step in the analytical process. Initially, the parent element is the

project. Subsequent data elements will then converge on core plug elements, which represent the

physical entities created in the laboratory and subsequently examined using microCT. The core plug

data elements that are used for quantitative microCT petrophysical analyses will typically correspond

to real world-entities known as sub plugs, which consist of 3 mm to 8 mm cores taken from physical

core plugs, commonly 25 mm in diameter. It is from these sub plugs (recorded in the database as core

plugs, often with larger-diameter core plugs as parent entities) that the automatically-harvested

metadata and image previews (of the tomographic images upon which the digital core analyses are

performed) are often attached.

A screenshot of the top-level page of the online database is shown below. Access to the data is

through either the search interface or the browsing interface; the mechanism of each approach is

elaborated in later sections of this guide.

Use this lin

k to access th

e

search interface U

se this link to access the

browsing interface

Data Elements are displayed as clickable hyperlinks to allow browsing

and editing in all contexts


User Guide / Page 3

ContentsOverview ..................................................................................................................................................... 4

Understanding DDCA – A Conceptual Framework ..................................................................................... 5

The Interface ............................................................................................................................................... 8

Browsing .................................................................................................................................................. 8

Searching ............................................................................................................................................... 12

Data Dictionary ......................................................................................................................................... 28

3D Visualisation ......................................................................................................................................... 29

Drishti ................................................................................................................................................ 32

Voluminous ....................................................................................................................................... 41


User Guide / Page 4

OverviewThis DDCA User Guide is developed with the expectation that there are three ways in which users

interact with the database (input, access, export) and two main types of users (analysts and

c l ients):

1. Analysts - typically engineers or laboratory staff associated with the ANU microCT laboratory or

FEI Lithicon - will primarily use the database to input information related to workflows as well as the

results of laboratory analyses or digital core analyses. Analysts might also exploit the hierarchical

and structured nature of the data – and the fact that the image metadata are automatically ingested

to the DDCA database – to enable easy access to analysis results.

2. C l ients – such as reservoir engineers, modellers, geologists and project managers – will access

the database for online viewing. Some will also export data (for example, tables of derived

properties such as porosity or permeability) for analysis or visualisation in other systems. This user

guide is aimed primarily at c l ients.

This guide covers three aspects of the database:

(1) The first section provides a general overview of the conceptual framework for DDCA.

(2) This is followed by an explanation of the interface, including the search functionality, as this is

the medium through which users will access the metadata within DDCA.

(3) The user guide then presents a data dictionary, which a user should understand in order to

successfully navigate the data within DDCA and modify it if necessary.

Further items, such as advice for 3D visualisation of tomographic data using external products such as

Drishti, are included in Appendix A.


User Guide / Page 5

UnderstandingtheDDCAThe DDCA (Database for Digita l Core Analys is) is designed to provide an integrated

environment for the access and management of the 3D datasets from microCT

analyses as wel l as the accompanying workf lows (for more detai l see Project 128

Milestone 2.9 Report) .

Figure 1: Overview of a typical workflow applied to core material

The purpose of the DDCA is to assist analysts and end-users to manage and understand the workflow,

that has been applied to core materials, of which an example is shown in Figure 1. The workflow

aspect of the database is facilitated by having a parent-child relationship between data elements.

Each element has a parent, and the parent chain will eventually lead to the parent project. Children of

each data element will generally represent subsequent steps in the digital and physical core analysis

workflow.

The data elements, shown in Figure 2, can be conceptually divided into two categories. The first

category covers the user-entered data elements that provide the contextual framework for the digital

core analysis as well as representing part of the physical workflow. That is, the first category

comprises the data elements (see the later section on the data structure of the database) that relate


User Guide / Page 6

to the origin of the core plug or sub plug that is eventually used in the microCT imaging process and

the digital core analysis. These data elements will typically include the PROJECT, the source of the

geological material (a WELL or OUTCROP), optionally a WHOLE CORE section and other information

relating to the WELL such as GEOLOGIC UNITS and WELL LOGS, and the CORE PLUG extracted from

the WHOLE CORE. The critical data element here is the CORE PLUG – and often there will be a chain

of CORE PLUGS representing cases where a sub plug has been extracted from a plug by physical

coring.

Figure 2: An overview of the DDCA Data Element Hierarchy

There is also a second suite of data elements relating to data that are automatically ingested

(harvested from archival data storage). This second category captures all of the data elements that

are not manually created by a user. Instead, they are generated through analytical processes and

form part of a rigid workflow in which various types of 3D images and derived entities (for example,

tomographic images, segmented images and pore network models) are obtained from parent image

acquisitions such as tomograms through an image analysis and computational pipeline. It is these data

elements that are of greatest interest to analysts and clients and it is from these data elements that

the derived properties such as porosity and permeability are calculated.


User Guide / Page 7

Project

Well

Well Log

Core Plug

Measured Properties

Whole Core Tomograms Segmented Images Pore-Networks SEM Images

Geological Unit Derived Properties

Outcrop Core Plug

F igure 3: A potential hierarchy of user-entered and automatical ly- ingested data, connected at the core-plug level and showing the ful l workflow for digital core analysis.

Coupling the user-entered data from the first category of elements to the automatically-ingested data

in the second suite of data elements (that is, the products of the imaging process) completes the

workflow chain, shown in Figure 3. This coupling generally occurs the CORE PLUG level since the core

plug is the physical entity to which the digital workflow is applied (keeping in mind that in the case of

sub plugs, CORE PLUGS can be entered as children of other CORE PLUGS ad infinitum), as shown in

the figure above.

Thus, the database allows digital core analysts to easily link the physical core specimens with the

imaging workflow they have undergone, and also to link analyses (for example, laboratory-based

measurements such as MICP) to the output of the micro-CT imaging process (this output is typically

the segmented data image data). End-users are therefore able to correlate petrophysical properties

calculated during digital core analysis with laboratory measurements and with representations of the

3D image data such as orthogonal slice images and 3D sub-sampled images.

Automatically-ingested data User-entered data


User Guide / Page 8

TheInterfaceThere are two approaches to accessing data within the DDCA Digital Database for Core Analysis:

browsing and searching. As shown in the Quick Start Guide, The browsing interface can be

accessed by clicking on the Projects link at the top right of the screen, inside the top panel that

features the DDCA logo. The search interface is located alongside and is labelled Search.

BrowsingAt all times, each data element can be

investigated by clicking on its name. This

reveals an interfaces that features the

type and name of the chosen element

and, under various panels, its re lat ions:

• Ancestors – that is, all of the

data elements in the hierarchy

above;

• Parent – the direct parent of

each elements;

• Sibl ings – elements of the same

type with the same parent;

• Children – data elements that

sit directly below the chosen

element;

• Descendants – all data elements that sit below the chosen element.

The relation panel (shown in Figure 4) allows a user to easily traverse the digital core analysis

workflow – each data element listed within is a clickable hyperlink which will take the user to a new

relation panel that focuses on the chosen element and lists the aforementioned relationships. A user

can therefore readily traverse the workflow chain simply by clicking on subsequent relations of a

chosen data element.

Below the relation panel, the DDCA interface displays an attr ibutes panel. This lists the details of the

chosen attribute – further information on what attributes correspond to what data type are shown in

Figure 4 screen shot showing par t of the relatio n panel


User Guide / Page 9

the following section, which describes the data dictionary. If the user has sufficient access privileges,

this panel features an EDIT button that allows the user to update or change the attribute data.

Finally, the interface features a custom data panel. Custom data may be of four possible types:

• string (text),

• number,

• Boolean (true/false),

• an attachment (a link to a file and a text description describing the file).

This allows for any data element to have supplementary information attached by users with edit

access rights. Importantly, the custom data follows a key-value system. For each entry of custom data

the user should specify the “key” for the chosen value. The key can be considered a label; consistent

use of labelling will facilitate searching and data export. For example, a measured property could be

attached to a core plug element by choosing the edit option under custom data, entering a key value

of Framework Grain Content %, a type of number and a value of 69.2, as shown in Figure 5.


User Guide / Page 10

Figure 5: Screen shot showing the attachment of a measured property to a core plug element after choosing the edit option under custom data



Users with the appropriate level of access can modify the structure of the data. As seen in Figure 6

they can delete the element being shown in the

browsing interface, create a child element (they will

be prompted to choose the type of the new child

element from a list of legitimate options), or adopt a

child.

Creating a new child element is straightforward as

seen in Figure 7 – the user is given the option of

specifying the TYPE of the new data element, and its NAME. The user can then follow the newly-

created hyperlink and, by using the EDIT button under the attributes subheading, populate the

relevant attributes.

Figure 7: Example of edit ing: the dialog box for adding a new chi ld data element

Figure 6 Screensho t sho wing how users with ‘edit ’ access can change the structure and con tent of the data

No data is ever truly deleted from the

DDCA and all operations are in principle

reversible. No user interface is yet

provided for undoing changes made in

error, this currently requires database

administrator assistance.



Adopting a child element allows the user to attach an element – and all its descendants – to a chosen

element as its child. The element being attached could be an orphan (not have a parent – typically

data that have been harvested before the matching user-generated data have been entered) or may

already be part of a chain of data elements. If so, it will be moved from its original chain – ALONG

WITH ALL OF ITS DESCENDANTS – and will be given the current element as its new parent.

The process of adopting an existing element (shown below) only requires one piece of information -

either the ID or URL of the child element which is to be adopted. The URL is usally readily obtained by

(in a Windows or Linux environment) right-clicking on the target child data element’s hyperlink within

the DDCA. The ID is the alpha-numeric sequence that forms the last part of the URL and it is generally

easier to supply the entire URL for this process.

Figure 8 Dialog for attaching an exist ing element to a new parent ( i .e. ‘adopting’ it)

All three operations – deleting an element, creating a child, adopting a child, manifest in the database

immediately.



SearchingUsers can use the search interface to find

specific data (based on its type, attributes

and more) and to display lists of data

elements with selected properties. The

search interface is accessed through the

search hyperlink at the right of the

topmost panel (featuring the DDCA logo;

see the Quick Start Guide at the beginning

of this document).

This section describes the search interface

(as shown in Figure 9) both in a general

sense and through the example of the

some of the pre-defined searches that are

stored within the DDCA (these are listed

below the search interface as hyperlinks).

Note, the results of a search process are

presented in a separate list (as shown in

the following sections). This list sits to the right of the search interface, except in cases where the

browser window is narrow. In these cases the search interface will reside in a query tab and the

results in a separate tab. Within the results list, each data element presents as a clickable hyperlink

that browses directly to the chosen data element in the same manner as the browse interface.

Figure 9 The toplevel searching dialo g, that expands as search terms are added



TheSearchElementsThere are five elements to the search interface:

1. the scope limits the search to all elements or makes the search relative to a chosen element

(specified by its URL), which allows the user to search the element’s ancestors or descendants

or children (recall, children are the direct descendant data elements, whereas descendants

extend all the way to the end of the data chain).

2. the data type option allows the user to limit the search results to a chosen data type (for

example, only core plugs). The user should use the option “imported-dataset” here if they

wish to only search among the automatically-ingested image data such as tomograms.

3. the condit ion panel provides limits on the search results,

4. the display panel allows the user to specify additional attributes to show in the results, and

5. Optionally, the user can control how many search results will be shown. The default is 50

records, and it is useful to maintain this value while developing a search, increasingly it only

when results are satisfactory. Showing many records (the options are 20, 50, 100, 200, 500

and 1000) may make the database timeout if the search query was poorly structured.

These search elements are discussed further below.

Scope:

This specifies whether a search extends through all data elements that the user can access (“entire

database”), or through the relations (ancestors, descendants, children) of a specific data element, a

specific data type, or elements with specified properties.

For example, to search only through the descendants of the acquisition dataset WW1_P22, one

would select "descendants of" and “base_element”, paste in the URL for WW1_P22 (https://ddca-

staging.anu.edu.au/entity/902c3ec8-7b39-4dfb-b6c1-fdc78718f8e9 - this can be right-clicking on the

link and selecting Copy Link Address in Chrome or Copy Link Location in Firefox) in the “ID or URL”

field. The "base element" itself is not included in the search. The results are shown in Figure 10.



Figure 10 Example of the results obtained from a search for al l descendents of a data element

The URL for

WW

1_P22 goes here



The scope functionality can also be used to easily search for relations of a particular data element –

for example, children of whole_core_section[s] shown in Figure 11.

Figure 11 Search results showing al l chi ldren of whole-core elements – showing the PU ‘upscal ing’ series



The scope functionality can also limit the search to relations of data elements with certain

characteristics, as in the case of the predefined search “SEM Registrations”. In this case the search is

limited in scope to descendants of elements where [attribute] name begins with “tomoSEM”, shown

in Figure 12.

Figure 12: Search example showing how to l imit the search results by adding Condit ions, in this case a constraint on the name



Data type:

Allows the user to indicate that the results should only show elements of the specified type. If "- any

data type –" is selected then all elements will be shown regardless of type. The effect of the data

type element is evident in the predefined search, “Well Search” shown in Figure 13

Figure 13 Search example: f ind al l data elements of a particular type (Well)



Note, all tomogram data are collectively harvested as imported_dataset types. So to view all

tomograms for, say, whole core sample CT1, one would search as shown in Figure 14 (note, the

search results have been truncated in the image shown):

Figure 14 Search example: f ind al l descendants of a particular element, identif ied by its URL (address)

The URL for CT1 goes

here



Condit ions:

This interface allows the user to specify additional conditions

for the search; only data elements that match the chosen

conditions will be returned by the search. The attribute field

auto-completes with a list of possible properties, so the user

does not need to know the exact spelling or data element

structure in order to complete a search.

For example, to look for an acquisition dataset with a sample

name that includes, say, the string "E23", one might type

"spec.sample.sample_name" under "attribute", then select "contains" from the drop-down menu and

finally type "E23" in the text box beneath it. Here, "spec.sample.sample_name" means something like

"take the value of the spec attribute, within that the value of the sample attribute, and within that,

the value of the sample_name attribute".

There are also limited wildcards for structured attributes. An asterisk (*.) at the beginning can stand

for any initial sequence of attribute names of dots. For example, "*.sample_name" would find

"spec.sample.sample_name", but also, say, something like "parameters.acquisition. sample_name" if

that existed. The asterisk only works at the beginning.

If the browser width is

insufficient the search interface

is automatically partitioned into

two panels, one featuring the

query and the other the results.



The example shown in Figure 15 returns core plugs between depths of 1000 and 1050 m and also

displays the depth attribute (the display option is described in the following section).

The condit ions search element is commonly used to pinpoint desired data from the database. See,

for example, the structure of the predefined “Acquisition Search” shown in Figure 16, which limits the

results to data of type “imported_dataset” with the condition that the process equals “Acquisition”

(note, this field is not case-sensitive):

Figure 15: search example: f ind a l l e lements o f a p art icular type wi th an attr ibute in a numer ical ran ge



Figure 16 Search example: (predefined “acquisit ion search”) f ind al l data elements where the ‘process’ that created them is named ‘acquisit ion’ – in this case the results are sets of projection data prior to tomographic reconstruction

Note, in order to explore potential options for an attribute condition, the user can include the desired

attribute as a display attribute from the matched element (the display search element is described

in the next section). An example of how that can be used to explore, for example, the range of values

for the “process” attribute is shown in Figure 17. From the results (not shown) it is evident that there

are numerous process types for imported datasets, including Acquisition, Clusters_Region_Merging,

Multi_Subset, Subset, Network_From_Labels, Registration_Resample and so on.

The condit ions element features eight states – equals, begins with, contains, is in range, does not

equal, does not begin with, does not contain, is not in range.



Figure 17 Search example: an attr ibute ( in this case ‘process’) can be included in the display f ield in order to get a feel for potential values, which can then be used in the condit ions f ield when searching.



D isplay:

The display options provide functionality for the user to specify which properties attributes should be

shown in the results table. Specifying attributes in this field allows the user to create a custom table

showing specific properties for the data elements listed in the search results. For example, Figure 18

shows a search process that returns core plugs and shows the diameter attribute in the results. As is

the case with the condition panel, the attribute field in the display panel auto-populates based on

possible values, so the user does not need to know the exact spelling or data element structure in

order to complete a search.

Figure 18 search example: the search interface showing a display attr ibute (diameter).



The display functionality is quite powerful – it allows for the results to be shown as a table or a

scatterplot (if the format of the resulting data allow it) and can show user-determined attributes from

either the “matched element” (that is, the elements found when the conditions in the scope, data

type and condit ions panels are met) as well as attributes from relations (ancestors, descendants,

children). These relation attributes can also refined based on the conditions of the chosen attribute in

the relation data element (possible conditional tests include equals, begins with, contains and so on –

the same eight as in the conditions panel).

Examples of the use of the display functionality are shown in the following section, which describes

a few of the predefined searches.



SearchingforSub-Plugs

One common search purpose would be to find sub-plugs. In the database, sub-plugs are stored as a

core-plug data element type (see the Data Dictionary in the following section), but will have core-

plugs as parent elements.

Parent attributes are not accessible in a search through the CONDITIONS field, however. To extract a

list of sub-plugs the user should modify the SCOPE of the search to include children of core plugs and

the returned data type as core plugs, as shown in Figure 19.

Figure 19 Searching for sub -plugs.



Exportingdata

The database provides an option to download

the search results in JSON or CSV format; the

DOWNLOAD option can be accessed below the

search panel as shown in Figure 20.

There is an additional CSV Template type for

users wishing to modify the data elements

returned in the search results. For example,

changing attribute values on all the returned

data elements, or perhaps creating child

elements from external data for the returned

data elements. For the CSV Template

download format the user is able to choose

which attributes are included in the

downloaded table. To actually modify the

DDCA data elements the user needs to convert

the CSV table into JSON data using a script

(external to the DDCA), before uploading the

JSON data at a separate page on the DDCA.

Figure 20: The search panel; no te the ‘Download Results ’ button



DataDictionaryThe DDCA interface automatically populates attributes field during searching, and lists the full suite of

properties during element editing. Nevertheless, it is useful for the user to have some understanding

of the data elements and their properties/attributes, as shown below for the user-generated data.

Project Outcrop WellLogName Elevationinm NameOrganisation Datum LogtypeProjectLeaders Surfacelocationin ResolutionNotes Latitude Units Longitude ScalePossibleChildren: Samplecollectedby ImageofLog-CaptionWell Sampleprovidedby ImageofLog-FileOutcrop Datesampled Env.Anddepthcorrection

Datesupplied Logdata-DepthWell Notes Logdata-ValueName NotesState PossibleChildren: Nameofbasin WholeCore CorePlugorSub-PlugField CorePlug NamePermit GeologicUnitorSub-Unit DiameterinmmENO/GAID LengthinmmPPDMID GeologicUnitorSub-Unit DepthinmPressureDBID Name PlugorientationStatus- GAStratigraphicunitID SedimentologicaldescriptionSpuddate DepthRange-From BoxnumberDatereachedTD DepthRange-To PlugPhoto-CaptionDaterigreleased Notes PlugPhoto-FileTotalDepth(driller’s) NotesDriller'sdatum PossibleChildren: TotalDepth(logger’s) GeologicUnitorSub-Unit PossibleChildren:Logger'sdatum MeasuredPropertyofPlugLocation WholeCoreSection CalculatedPropertyElevationinm Name ImportedDatasetDatum UnitID SEM-EDSImageSurfacelocationin Sub-UnitID SEMImageLatitude Diameterinmm Longitude Lengthinmm MeasuredPropertyofPlugCompany Depthinm NameTypeofwell Sedimentologicaldescription PhysicalpropertyBitsizeinmm Intervalinm-Top UnitsPreservedcore Intervalinm-Bottom ValueNotes CorePhoto-Caption LabTechnique…Preservedcore CorePhoto-File LabIDNotes Notes Date NotesPossibleChildren: PossibleChildren: WellLog CorePlug WholeCore GeologicalUnitorSub-Unit CorePlug KeytoPropertyFormats:TextNumberDate



3DVisualisationThe DDCA is primarily a system for managing metadata, curating data and representing workflows.

High-performance visualisation of the data, particularly in 3D, is beyond the scope of the database

itself.

There is, however, potential for industry-standard visualisation of data stored in the DDCA using freely

available tools developed at the Australian National University, both in the Department of Applied

Mathematics and in the National Computational Infrastructure (NCI) VizLab. This software-

development work has occurred in concert with developments at the ANU x-ray micro-CT imaging

facility.

These tools are Drishti and Voluminous. As well as the 3D visualisation that these tools offer, users

can interact with large 3D datasets in a 2D slice-wise manner, or overlay individual slices, using the

NCViewer slice viewer (developed by the Australian National University and to be made available to

users of the DDCA). This section provides an overview of each tool and provides a more in-depth

explanation of the use of Drishti, which is routinely used for 3D visualisation in a commercial

production environment. By using Drishti, DDCA users will therefore have access to industry-standard

visualisation capabilities.

Down-scaledvolumesforvisualisation

Volumetric data require considerable space for storage and processing power for visualisation. The

DDCA therefore provides access to downscaled raw tomograms and segmented images for use in 3D

visualisation, as well as 2D slices for use when the user is not able to use 3D methods.

Voxels in volumes from µCT are often stored as 16-bit integer values, with the number of voxels in

each dimension being a function of the sample size (diameter and height) and the detector

configuration and capabilities. Within a volume, each voxel requires two bytes to store and so file

sizes are slightly more than the product of the X, Y, Z dimensions multiplied by two (the metadata

being the reason that the file sizes are “slightly more” than size of the raw image data). A typical

image, of 1760 x 1760 x 3500 voxels, therefore measures around 20 GB, which is not amenable to

visualisation on anything but the highest performance computer hardware. Nor can such volumes be

downloaded over the internet within any reasonable period. In addition, such images of complex

geomaterials usually contain too much detailed information to be visualised in three dimensions.



Therefore, the DDCA provides downscaled versions of volumes – in the form of raw µCT and

segmented images – for user visualisation. The volumes are downscaled four-to-one in each

dimension, usually resulting in 480 voxels in the X and Y dimensions (with the Z dimension remaining

a function of the sub-plug height). These volumes will be stored in their native netCDF file format,

which can be readily imported into Drishti, and NCViewer. For import into Voluminous the user will

need to use the Drishti import tool to convert the netCDF file into an MHD (meta-image) file set.

The system will also provide tiff (tagged image file format) slices for volumes, which can be viewed in

NCViewer, but are generally not as useful for interpretation as the 3D volumes viewed in either

NCViewer or Drishti.

NCViewer

NCViewer provides interactive visualisation of 3D volumes stored in netCDF and is routinely used in a

production environment to explore 3D dataset, choose values for processing parameters (for

example, thresholds for segmentation) and investigating the outcomes of processing operations.

NCViewer is freely available on request from ANU.

The software allows a user to traverse through stacked layers of a 3D tomogram or segmented image.

NCViewer only provides 2D representations of the data, though it is important not to underestimate

the importance of this tool for quantitative analysis of images. Firstly, the mapping between voxel

values intensity is well defined, so that one can see the actual data values. Secondly, one can overlay

images upon one another, allowing comparison between original and processed images. Thirdly, one

can zoom in until the values of individual voxels are evident. Finally, one can test the results of

applying thresholds or conduct measurements of feature size or contrast and image noise levels. For

these reasons, NCViewer, rather than 3D visualisation is an integral part of the quantitative analysis

workflow in a production environment. In particular, NCViewer allows the user to determine the

thresholds and gradients that will be used for image filtering and segmentation; critical steps in

creating binarised data from which calculated properties (for example, porosity, permeability, pore

networks) are derived.

For example, as Figure 21 shows, one can interactively change the thresholds, and the colours

assigned to them, on the slice histogram. It also allows users to manually paint tomograms, calculate

statistics on particular regions of the tomogram, measure distances, and export the tomogram as a

series of images.



As with visualisation via Drishti, a user can export the downscaled netCDF data from the DDCA and

readily import it into NCViewer for exploration, as shown in Figure 21.

Figure 21 The NCViewer interface, shown h ere depict ing layer 61 from a tomo gram im age block .



Drishti

Drishti is cross-platform desktop software for volume rendering developed by Ajay Limaye at the NCI

Vizlab and released under the GNU GPL v3 open software licence. Drishti, which is freely available via

download from google code, has been in development for over 12 years and is a stable, mature

product with an active user base of over 1,000 users. It is routinely used in a commercial production

environment for the visualisation of µCT tomograms and segmented (Mango-processed) images.

Designed for the exploration of volumetric datasets, it allows users to import image or volume files,

downscale them as required and then render up to four volumes up to 1024 x 1024 x 1024,

depending on the graphics hardware on the host computer. The user is able to interactively

manipulate the volume, rotating, zooming and displacing it, altering its colour, shading and

transparency, and taking slices in any axis and on any angle as well as exporting all views as static

images or movies.

CT or MANGO image is uploaded into DDCA

DDCA creates downscaled

version (4-to-1 in each

dimention)

User downloadeds downscaled

version (netCDF) to

desktop

User imports netCDF into Drishti (or

NCViewer) for visusalisation

Figure 22 Gener ic workf low for 3D vi sua lisat ion of data from th e DDCA



Figure 23 3D Visual isation of a downscaled tomogram and segmented volume using Drishti software.

Figure 24 The Drisht i import package a l lows a netCDF f i le from the DDCA to be adjusted b efore 3D vi sua lisat ion.



UsingDristhi2.4forvisualisationofDDCAdata

Interested users can gain a general understanding of the use of Drishti through the online user guide,

help videos and Drishti User Group, all of which are accessible from the Google Code page on which

Drishti is hosted: https://code.google.com/p/drishti-2/

This section describes the basic steps involved in visualising a downscaled tomogram from the DDCA

into Drishti, paying particular attention to the steps that are unique to data from digital core analyses.

Specifically, it covers the visualisation of segmented images, which have discrete, rather than

continuous, histograms and therefore require a different approach for shading than regular

tomograms. For example, Figure 25 Figure 26shows a raw µCT tomogram image – note the

continuous histogram (highlighted in red) – whereas the histogram for a segmented image shows

discrete intervals and requires a separate transfer function for each one (Figure 26).

The general process for visualisation a downscaled volume from the DDCA in Dhristi is:

(1) download the downscaled netCDF volume (either a raw tomogram or a segmented volume)

from the DDCA. Typically, the segmented image is shown together with the raw tomogram,

so both volumes should be opened in turn.

(2) open the first volume (either the raw tomogram or segmented image, the order is not

important) using the Drishti Import program:



(3) Adjust the histogram to achieve maximum contrast in the image – this step is critical as it

will change the values in the exported volume:



(4) Save the processed volume as a Drishti image (*.pvl.nc) – access all options that arise during

the export process in their default state.

(5) Repeat the process for the second volume (to ensure that both the raw tomogram and the

segmented image have been ‘imported’).

(6) Close Drishti Import and open Drishti.

(7) Load both volumes together, as shown below.



(8) The results are shown below:

(9) Now create more transfer functions (the Transfer Function Editor is located in the top right

of the image above) to capture the discrete distribution of the segmented image – one

transfer function is needed for each ‘peak’.

(10) Ensure that the tick box for the new transfer functions are in the column that corresponds

to the segmented image (depending on what order it was loaded).



(11) Adjust the histograms for the raw tomogram and the segmented image – the segmented

image should have one transfer function on each peak in the histogram, as shown below

(note the green transfer function, which coincides with the first peak in intensity in the

segmented volume):



Figure 25 3D Visua lisat ion of a down sca led tomo gram for W est W andoan sam ple P12. TF (Transfer Fun ctio n) 0 corresponds to the raw tom ogram. TF 1, 2 , and 3 co rrespond to the segmented image and a shown inFigure 6. Note the d iscrete distribut ion of the segmented image histograms.



Figure 26 3D Visua lisat ion of a downscaled segmented volum e for West Wandoan sample P12. Th ree transfer fu nctio ns have been created to capture th e three discrete sect io ns of m ateria l in the image histogram; transfer funct ion 1 is vis ibly act ive and covers the fi rst sectio n in the histogram; transfer funct ions 2 and 3 (n ot vi sible) have been ass igned di fferent colou rs and cover the next two discrete gro upings in the h istogram (marked by arrows).



Voluminous

The Voluminous 3D visualisation software is being developed by the National Computational

Infrastructure (NCI) VizLab at the Australian National University. Voluminous may one day have the

capability to accommodate the visualisation of full-scale netCDF data within the DDCA, given

sufficient rendering hardware on the DDCA web server.

Voluminous is remote volume-rendering software deployed as a cloud service. Data are stored and

visualisations rendered on a remote server with high-performance graphics capabilities. Users can

access and manipulate the visualisations in real time via their web browser (Figure 27).

Cloud-based storage and visualisation presents potential difficulties related to security and bandwidth

as files currently need to be uploaded to the remote Voluminous server. Security concerns are

ameliorated by the fact that the data from Mango will already reside on ANU secure servers. Ideally,

Voluminous will be deployed on the same server as the DDCA, so that the large data sets would not

need to be moved. In this configuration, users would identify a dataset for visualisation from the

DDCA environment, then be transferred to Voluminous, most likely by opening a new tab in their

browser. The data would immediately be available for visualisation with Voluminous. While this

behaviour is part of the long-term roadmap for Voluminous, it is not available today nor part of short-

term plans. In the meantime, users will need to export data from the DDCA then upload it for

visualisation with Voluminous.



Figure 27 Visua lisat ion of data from the DDCA us ing the cloud-based system, Vo luminous

submitted as milestone 2.10 under anlec project 0128 sub...

Documents