productive & sustainable: more effective cse

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. !

Productive & Sustainable:More Effective CSEMike HerouxSenior ScientistCenter for Computing ResearchSandia National Laboratories

Scientist in ResidenceSt. John’s University, MN

MichaelHerouxSIAMCSE2017 1

16thSIAMConferenceonParallelProcessingforScien>ficCompu>ng(PP18)March7-10,2018,Tokyo,Japan

§  Venue§  Nishi-WasedaCampus,WasedaUniversity

§  OrganizingCommiSeeCo-Chairs’s§  SatoshiMatsuoka(TokyoIns>tuteofTechnology,Japan)§  KengoNakajima(TheUniversityofTokyo,Japan)§  OlafSchenk(UniversitadellaSvizzeraitaliana,Switzerland)

§  Contact§  KengoNakajima,nakajima(at)cc.u-tokyo.ac.jp§  hSp://nkl.cc.u-tokyo.ac.jp/SIAMPP18/§  http://www.siam.org/meetings/pp18/


TEAMS,COLLABORATORS,INSPIRATIONS

3MichaelHerouxSIAMCSE2017

DOEASCRIDEASProduc>vityProject


CollaboratorsandInfluencers

5

Michael Heroux (SNL), Co-Lead PI Lois Curfman McInnes (ANL), Co-Lead PI David Bernholdt (ORNL), Co-PI, Outreach Lead Todd Gamblin (LLNL), Co-PI Osni Marques (LBNL), Co-PI David Moulton (LANL), Co-PI Boyana Norris (Univ of Oregon), Co-PI Satish Balay (ANL) Roscoe Bartlett (SNL) Anshu Dubey (ANL) Rinku Gupta (ANL) Christoph Junghans (LANL) Alicia Klinvex (SNL)

Reed Milewicz (SNL) Mark Miller (LLNL) Elaine Raybourn (SNL) Barry Smith (ANL) Louis Vernon (LANL) Greg Watson (ORNL) James Willenbring (SNL) Lisa Childers (ALCF) Rebecca Hartman-Baker (NERSC) Judy Hill (OLCF) Hai Ah Nam (LANL) Jean Shuler (LLNL)

Facilities Liaisons

IDEAS-ECPProduc>vityProject

+xSDK-ECPProjectMichaelHerouxSIAMCSE2017

Inspira>onsandCollabora>ons


Outline

§  Concepts&Defini>ons.§  WhyIcareaboutproduc>vity&sustainability.§  Incen>ves:BeSerproduc>vity&sustainability.§  Completenessphilosophy.§  SomePrac>calStrategies.§  Produc>vity++andBeSerScien>ficSoeware

7 MichaelHerouxSIAMCSE2017 7

MyRoles

§ Regardingobserva>onsonopportuni>estoimprove:§ Morelikeapsychologistthanexpert.

§ Regardingsoewaretools,processes,prac>cesimprovements:§ Morelikeacarpenterthanexpert.


CONCEPTS&DEFINITIONS


Produc>vity&Sustainability

§  Produc>vity–Outputperunitinput.§  Specifically:DeveloperProduc9vity.

§  Sustainability–Thefuturecostofusability.§  Specifically:So;wareSustainability.

§  Strategy:Focusonimprovement.§  Optometristapproach.


Scenario:Youarebehindindevelopingasophis9catednewmodelinyourso;warethatyouwanttouseforresultsinanupcomingpaper.Whichofthesecouldbereasonablechoices?

§  Developasimplermodelforthepaper.§  Setotherworkasideandspendmore>meondevelopment.§  Askforanextensiononthepaperdeadline.§  Developsophis>catedmodel,butdon’ttestitscorrectness.§  Developsophis>catedmodel,butdon’tdocumentitorcheckitin.

BeSer,Faster,Cheaper:Picktwoofthethree.


ImprovedDeveloperProduc>vity

“BeSer,Faster,Cheaper:Pickallthree.”–Nearterm.Scenario:(6monthslater)Aeerinves>ngindeveloperproduc<vityimprovements,youareon>meindevelopingasophis>catednewmodelinyoursoewarethatyouwanttouseforresultsinanupcomingpaper.Investindevelopertools,processes,prac>ces.


ImprovedSoewareSustainability

“BeSer,Faster,Cheaper:Pickallthree.”–Longterm.Scenario:(3yearslater)Aeerinves>nginsoCwaresustainabilityimprovements,youareon>meindevelopingseveralsophis>catednewmodelsinyoursoewarethatyouwanttouseforresultsinupcomingpapers.Investintes>ng,documenta>on,integra>onforlong-termsoewareusability.


PRODUCTIVITY&SUSTAINABILITY:WHYNOW?


Reproducibility

• NY Times highlights “problems”.

• Only one of many cited examples.

• CSE has been spared this “spotlight” (so far).

http://www.nytimes.com/2015/08/28/science/many-social-science-findings-not-as-strong-as-claimed-study-says.html?_r=0!


FutureHighPerformanceCompu>ng:§ ISNOTprimarilyaboutHW.§ ISprimarilyaboutAlgorithms.§ ISprimarilyaboutsoeware:

§ ΔA,ΔS>>ΔH§ Algs:AdequateR&Dfocus.§ SW:Opportuni>esforimprovement.


“Easy”WorkinProgress§  Newparallelalgorithms:

§  Turningouttobefeasible.§  Cleverideas:DavidKeyestalkfromMonday.§  Lotstodo,butsteadyprogress§ Muchevidenceinthisconference.

§  CurrentThreadProgrammingEnvironments:§  C++,OpenMP,others:Working.§  Run>mes:S>llalotofwork,butprogress.

§  Lotstodo,butcommunityisfocused.


“Hard”Work

§  BillionsSLOCofencodedscience&engineering.

§  Challenge:§  Transfer,refactor,rewriteformodernsystems.§  Dosowithmodestinvestmentbumpup.§ Workacrossdis>nctanddiverseteams.§  Deliverscienceatthesame>me.§ Makethenextdisrup>oneasiertoaddress.


TOWARDREPRODUCIBLESCIENCE:CREATINGINCENTIVES


Incen>vesToChange

•  Reproducibility •  SW Quality Requirements •  Employer Recognition

Productivity & Sustainability Investments

Demand

Enable Common statement: “I would love to do a better job, but I need to: •  Get this paper submitted. •  Complete this project task. •  Do something my employer values more. Goal: Change incentives to include value of better software.


ReproducibilityTerminology

§  ReviewableResearch.Thedescrip>onsoftheresearchmethodscanbeindependentlyassessedandtheresultsjudgedcredible.(Thisincludesbothtradi>onalpeerreviewandcommunityreview,anddoesnotnecessarilyimplyreproducibility.)

§  ReplicableResearch.Toolsaremadeavailablethatwouldallowonetoduplicatetheresultsoftheresearch,forexamplebyrunningtheauthors’codetoproducetheplotsshowninthepublica>on.(Heretoolsmightbelimitedinscope,e.g.,onlyessen>aldataorexecutables,andmightonlybemadeavailabletorefereesoronlyuponrequest.)

§  ConfirmableResearch.ThemainconclusionsoftheresearchcanbeaSainedindependentlywithouttheuseofsoewareprovidedbytheauthor.(Butusingthecompletedescrip>onofalgorithmsandmethodologyprovidedinthepublica>onandanysupplementarymaterials.)

§  AuditableResearch.Sufficientrecords(includingdataandsoeware)havebeenarchivedsothattheresearchcanbedefendedlaterifnecessaryordifferencesbetweenindependentconfirma>onsresolved.Thearchivemightbeprivate,aswithtradi>onallaboratorynotebooks.

§  OpenorReproducibleResearch.Auditableresearchmadeopenlyavailable.Thiscomprisedwell-documentedandfullyopencodeanddatathatarepubliclyavailablethatwouldallowoneto(a)fullyauditthecomputa>onalprocedure,(b)replicateandalsoindependentlyreproducetheresultsoftheresearch,and(c)extendtheresultsorapplythemethodtonewproblems.

V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider, and W. Stein. 2013. Setting the Default to Reproducible: Reproducibility in

Computational and Experimental Mathematics. (2013). https://icerm.brown.edu/tw12-5-rcem/icerm_report.pdf


§  TOMSRCRIni>a>ve:RefereeData.§ WhyTOMS?

§ Tradi>onofrealsoewarethatothersuse.

ACMTOMS


ACMTOMSReplicatedComputa>onalResults(RCR)

§  Standardreviewerassignment:§  Nothingchanges.

§  RCRreviewerassignment:§  Concurrentwithstandardreviews.

§  RCRprocess:§ Mul>-facetedapproach,BoSomline:Trustthereviewer.

§  Publica>on:§  ReplicatedComputa>onalResultsDesigna>on.§  Reviewreportappearswithpublishedmanuscript.


RCRProcess:TwoBasicApproaches

1.  Independentreplica>on:§  Openreview.

2.  Reviewofcomputa>onalresultsar>facts:§ ”Bou>que”compu>ngsystem.§  Inthissitua>on:

§ Carefuldocumenta>onoftheprocess.§ Soewareshouldhaveitsownsubstan>alV&Vprocess.


BigPictureofTOMSRCR

§  Improvescience.

§  Notgoodnow:§  Trustcomesfrom“cloud”ofpaperswithsimilarresults.§ Whichcoulds>llbewrong.§  Replicability:Firststeptowardimprovement.

§  Engagea“darkpor>on”oftheR&Dcommunity.§  Reviewersnotamongtypicalreviewerpool.§  Prac>>oners,users.ExpertatuseofMathSW.

Thank you for taking the time to consider our paper for your journal. XXX has agreed to undergo the RCR process should the paper proceed far enough in the review process to qualify. To make this easier we have preserved the exact copy of the code used for the results (including additional code for generating detailed statistics that is not in the library version of the code).


ComingtoYourWorldSoon:ReproducibilityRequirements

§  Theseconferencesexpectar>factevalua>onappendices(mostop>onally):§  CGO,PPoPP,PACT,RTSSandSC.§  hSp://fursin.net/reproducibility.html

§  ACMReplicatedComputa>onalResults(RCR).§  ACMTOMS,TOMACS.§  hSp://toms.acm.org/replicated-computa>onal-results.cfm

§  ACMBadging.§  hSps://www.acm.org/publica>ons/policies/ar>fact-review-badging


SC17ReproducibilityIni>a>ve§  Twoappendices:

§ Ar>factdescrip>on(AD).§ Schema>cofyourcompu>ngenvironment.§ Makesiteasiertoreruncomputa>onsinfuture.§ ADrequiredforbestpaper&studentpaper.

§ Computa>onalResultsAnalysis(CRA).§ Targets”bou>que”environments.§ Improvestrustworthiness.

§ Details:hSp://sc17.supercompu>ng.org/submiSers/technical-papers/reproducibility-ini>a>ves-for-technical-papers/


SourcesforCRAmetrics§  Synthe>coperatorswithknown:

§  Spectrum(Hugediagonals).§  Rank(byconstruc>ons).

§  Invariantsubspaces:§  Example:Posi>onal/rota>onalinvariance(structures).

§  Conserva>onprinciples:§  Example:Fluxthroughafinitevolume.

§  General:§  Pre-condi>ons,post-condi>ons,invariants.


Incen>ves&Produc>vity/Sustainability

•  Reproducibility •  SW Quality Requirements

•  Employer Recognition

SW Productivity & Sustainability Investments

Demand

Enable

New funding proposal element: SW Productivity and Sustainability Plan


DOESWProduc>vityandSustainabilityPlan(SWPSP).

§  KeyEn>>es:§  DOEBiologicalandEnvironmentalResearch(BER).§  DOEAdvancedScien>ficCompu>ngResearch(ASCR)§  IDEASProject

§  Milestone:§  First-of-a-kindSWProduc>vityandSustainabilityPlan.


DOEBERSWPSPRequirements

§  DescribeoverallSWdevelopmentprocess.§  Soewarelifecycle,tes>ng,documenta>onandtraining.

§  Developmenttoolsandprocesses:§  sourcemanagement,issuetracking,regressiontes>ng,SWdistribu>on.

§  Trainingandtransi>on:§  Newanddepar>ngteammembers.

§  Con>nuousprocessimprovement:§  GetngbeSeratproduc>vityandsustainability.


Employers&Produc>vity/Sustainability

•  Reproducibility •  SW Quality Requirements •  Employer Recognition

SW Productivity & Sustainability Investments

Demand

Enable

Next focus: •  Work with DOE labs to recognize SW as base research.

•  Pressing for Scientific SW Productivity R&D base funding.


HowtheFuturewillbeDifferent

§  Publishers:§  Willexpectreproduciblecomputa>onalresults.

§  Fundingagencies:§  Willexpectimprovedproduc>vity,sustainablesoeware.

§  Employers:§  Willrewardstaff,facultyproducinggoodsoeware.

§  Impact:High-qualityCSEsoewarewillmaSeralotmore.


PRODUCTIVITYIMPROVEMENT


CSE&FormalSoewareMethodologies:TroubledHistory

35

§  Cray(1990):§  FormalWaterfallMethod.

§  DOEASCI(2000):§  CMMI

§  Failedtofollowownprocess:Elicitrequirements.

MichaelHerouxSIAMCSE2017

CSEComplete:Useful“Overhead”§  CodeComplete:Ul>matevalueiscode.

§  Shouldweonlywritecode?§  Somenon-codingac>vi>esimprovecode.

§  CSEComplete:Ul>matevalueisCSE.§  Whatac>vi>esimproveCSE?

36

“Plans are worthless, but planning is everything.” Dwight D. Eisenhower

“Give me six hours to chop down a tree and I will spend the first four sharpening the axe.”

Abraham Lincoln


PRODUCTIVITYIMPROVEMENT:PLANNING


38

Visible Progress (Writing code, computing results)

100 %

0 % Planning

Recoding and Porting to new Platforms

Percent Effort

Time Midlife Effort Profile

Endlife Effort Profile

Adapted from Software Project Survival Guide, Steve McConnell

Code-and-Fix Development Approach

Early Effort Profile


39

Visible Progress (Writing code, computing results)

100 %

0 %

Planning

Recoding and Porting to new Platforms

Percent Effort

Time

Simple Planned Development Approach

Midlife Effort Profile

Ongoing Effort Profile

Early Effort Profile


Planningtools:Usewhatyouknow

40

KokkosKernels - add design note for discussion. This design note is very informal and working note for discussion. … For typeset, "make"

KokkosKernels - add more algorithm variants. In this algorithm design, I have a few assumptions. …

Commit log messages Latex Planning Document

KokkosKernels:Micro&BatchedBLASDesignDocument§  6weeks:DesignbyLaTeX.

§  Reviewbydiverseexperts.§  Significantdesignchanges:Intextonly.

§  2weeks:Writecode.

Message:Usethetoolsyouknow.

Courtesy: KokkosKernels Development Team


PRODUCTIVITYIMPROVEMENT:ISSUETRACKING


Managingissues:Fundamentalsoewareprocess

42

§  Issue:Bugreport,featurerequest§  Approaches:

§  Short-termmemory,officenotepad§  ToDo.txtoncomputerdesktop(1person)§  Issues.txtinrepositoryroot(smallco-locatedteam)§  …§ Web-basedtool+Kanban(distributed,largerteam)§ Web-basedtool+Scrum(full->medevteam)

Informal, less training

Formal, more training

Continual improvement


Kanbanprinciples

43

§  Limitnumberof“InProgress”tasks§  Produc>vityimprovement:

§  Op>mize“flexibilityvsswapoverhead”balance.Noovercommitng.

§  Produc>vityweaknessexposedasboSleneck.Teammustiden>fyandfixtheboSleneck.

§  Effec>veinR&Dsetng.Avoidsadeadline-basedapproach.Deadlinesaredealtwithinadifferentway.

§  Providesaboardforviewingandmanagingissues


BasicKanbanBacklog Ready In Progress Done •  Any

task idea

•  Task + how to do it.

•  Could be pulled when slot opens.

•  Task you are working on right now.

•  The only Kanban rule: Can have only so many “In Progress” tasks.

•  Limit is based on experience, calibration.

•  Completed tasks.

•  Record of your life activities.

•  Rate of completion is your “velocity”.


Kanban Iterative Develop

ment

X Loop

KanbanX

X=Sustainability,Performance

TOWARDAREPEATABLEPROCESS


SummarizeProjectPrac>ces

SetGoals ConstructPTC

RecordCurrentPTC

Values

CreateImprovement

Plan

Repeat

AssessProgress

PSIPWorkflow

ExecutePlan

Start

SWProduc>vity&SustainabilityImprovementPlans


ProgressTrackingCardExample


Practice: Test Coverage Score (0 – 5): Score Descriptions 0 Little or no independent testing. Functional testing via users. 1 Independent functional testing of primary capabilities. 2 Primary functional testing, some unit test coverage. 3 Comprehensive unit testing, primary functional testing. 4 Comprehensive unit testing, functional testing for documented use cases. 5 Comprehensive unit, use case functional testing; test coverage commitment.

49

BSS Software Platform

Component Technology

Backend Frontend

Google Docs GitHub Ruby on Rails

Location Google Drive betterscientificsoftware organization

betterscientificsoftware.info

Purpose •  Rapid collaborative content development.

•  Multi-user typing, suggested edits, comments.

•  Content creation, refinement, management (from Google Drive).

•  Content packaging for use with BSS.info

•  User-facing portal •  Polished backend

content •  Blogs, forums •  Mailing lists.

Contributors Community content experts

Community content experts. BSS staff.

BSS staff. Web dev experts.

Consumers BSS GitHub Backend BSS Frontend CSE community

Content Notes

Content migrates to GitHub after it stabilizes.

Content is managed in git repos, markdown.

Content from backend, community.

New community portal: Scientific Software Improvement Exchange https://betterscientificsoftware.io May

2017


Summary§  CSEisnotcompletewithoutincorpora>ngbeSer:

§  Reproducibility:Canyourresultsbetrusted?§  Sustainability:Willyourecosystemlivelongandprosper?§  Produc>vity:Willyoursoewaredevelopmenteffortsbecompe>>ve?

§  Produc>vityeffortmustenhanceCSEresults:§  Highquality,lowvaluesoewarecannotbetheoutcome.§  Qualityimprovementsmustbemeasured.

§  Changesinpublishers,funders,employersexpecta>ons:§  Couldviewasbothersome:More“overhead”impedingprogress.§  Couldbeopportunity:Beproac>ve,partoftheini>a>ve.


Produc>vity++Ini>a>veAsk:IsMyWork_______?

https://github.com/trilinos/Trilinos/wiki/Productivity---Initiative

Productivity++ Traceable In Progress Sustainable Improved

Version(1.2(


Contribute!

§  hSps://github.com/beSerscien>ficsoeware/beSerscien>ficsoeware.github.io/blob/master/README.md

§  Orsearch“githubbeSerscien>ficsoeware”.


Acknowledgments53

§  SandiaNa>onalLaboratoriesisamul>-programlaboratorymanagedandoperatedbySandiaCorpora>on,awhollyownedsubsidiaryofLockheedMar>nCorpora>on,fortheU.S.DepartmentofEnergy’sNa>onalNuclearSecurityAdministra>onundercontractDE-AC04-94AL85000.SANDNO.2016-8466C.


PRODUCTIVITYIMPROVEMENT:APPLICATIONCOMPOSITIONANDSOFTWAREECO-SYSTEMS


Extreme-ScaleScien<ficSoCwareEcosystem

Libraries•  Solvers,etc.•  Interoperable.

Frameworks&tools•  Docgenerators.•  Test,buildframework.

Extreme-ScaleScien<ficSoCwareDevelopmentKit(xSDK)

SWengineering•  Produc>vitytools.•  Models,processes.

Domaincomponents•  Reac>ngflow,etc.•  Reusable.

Documenta>oncontent•  Sourcemarkup.•  Embeddedexamples.

Tes>ngcontent•  Unittests.•  Testfixtures.

Buildcontent•  Rules.•  Parameters.

Libraryinterfaces•  Parameterlists.•  Interfaceadapters.•  Func>oncalls.

Shareddataobjects•  Meshes.•  Matrices,vectors.

Na>vecode&dataobjects•  Singleusecode.•  Coordinatedcomponentuse.•  Applica>onspecific.

Extreme-scaleScienceApplica<ons

Domaincomponentinterfaces•  Datamediatorinterac>ons.•  Hierarchicalorganiza>on.•  Mul>scale/mul>physicscoupling.


xSDKfocus

56§  Commonconfigureandlinkcapabili>es

§  xSDKusersneedfullandconsistentaccesstoallxSDKcapabili>es§  Namespaceandversionconflictsmakesimultaneousbuild/linkof

xSDKdifficult§  Determininganapproachthatcanbeadoptedbyanylibraryor

componentsdevelopmentteamforstandardizedconfigure/linkprocesses

§  Libraryinteroperability§  Designingforperformanceportability

Libraries•  Solvers,etc.•  Interoperable.

Frameworks&tools•  Docgenerators.•  Test,buildframework.

Extreme-ScaleScien>ficSoewareDevelopmentKit(xSDK)

SWengineering•  Produc>vitytools.•  Models,processes.

Domaincomponents•  Reac>ngflow,etc.•  Reusable.


StandardxSDKpackageinstalla>oninterface

57

Motivation: Obtaining, configuring, and installing multiple independent software packages is tedious and error prone. !•  Need consistency of compiler (+version,

options), 3rd-party packages, etc.!Approach: Define a standard xSDK package installation interface to which all xSDK packages will subscribe and be tested !Accomplishments: !•  Work on implementations of the standard by

the hypre, PETSc, SuperLU, and Trilinos developers!

•  PETSc can now use the “scriptable” feature of the installers to simultaneously install hypre, PETSc, SuperLU, Trilinos with consistent compilers and ‘helper’ libraries.! Impact: Foundational step toward

seamless combined use of xSDK libraries, as needed by BER use cases and other multiphysics apps!

!

xSDK Build Example!


xSDKMinimumComplianceRequirements:

58

§  M1.EachxSDKcompliantpackagemustsupportthethestandardxSDKcmake/configureop>ons.

§  M2.EachxSDKpackagemustprovideacomprehensivetestsuitethatcanberunbyusersanddoesnotrequirethepurchaseofcommercialsoeware

§  M3.EachxSDKcompliantpackagethatu>lizesMPImustrestrictitsMPIopera>onstoMPIcommunicatorsthatareprovidedtoitandnotusedirectlyMPI_COMM_WORLD.

§  M4.Eachpackageteammustdoa‘besteffort’atportabilitytokeyarchitectures,includingstandardLinuxdistribu>ons,GNU,Clang,vendorcompilers,andtargetmachinesatALCF,NERSC,OLCF.AppleMacOSandMicrosoeWindowssupportarerecommended.

§  M5.Eachpackageteammustprovideadocumented,reliablewaytocontactthedevelopmentteam;thismaybebyemailorawebsite.Thepackageteamsshouldnotrequireuserstojoinagenericmailinglist(andhencereceiveirrelevantemailtheymustwadethrough)inordertoreportbugsorrequestassistance.

§  M6–11…

https://ideas-productivity.org/resources/xsdk-docs MichaelHerouxSIAMCSE2017

xSDKRecommendedComplianceRequirements:

59

§  R1.Itisrecommendedthateachpackagehaveapublicrepository,forexampleatgithuborbitbucket,wherethedevelopmentversionofthepackageisavailable.Supportfortakingpullrequestsisalsorecommended.

§  R2.Itisrecommendthatalllibrariesbetestedwithvalgrindformemorycorrup>onissueswhilethetestsuiteisrun.

§  R3.Itisrecommendedthateachpackageadoptanddocumentaconsistentsystemforpropaga>ng/returningerrorcondi>ons/excep>onsandprovideanAPIforchangingthebehavior.

§  R4.Itisrecommendedthateachpackagefreeallsystemresourcesithasacquiredassoonastheyarenolongerneeded.


MessagetoThisAudience

Considerwhatso;wareecosystem(s)youwantyourso;waretobepartofanduse.


SUSTAINABILITYIMPROVEMENT


“Alwayscodeasiftheguywhoendsupmaintainingyourcodewillbeaviolentpsychopathwhoknowswhereyoulive.”

-JohnWood


Sustainable?Pullrequestscenario

• Pull request. • Competent external user. • Useful feature. • No conflicts. • Accept: Yes, if OK. • What must happen?

• Vet commit: –  Comes with tests. –  Comes with docs. –  Follows style rules.

•  Test: –  No regressions. –  Portable.


MessagetoThisAudience

Writetestsnow,while(orbefore)wri9ngyourintendedproduc9onso;ware.


productive & sustainable: more effective cse

Documents