(a.k.a “damns”, “dambs”, “damu”, “daaamms”) dams · a brief history of the fmnh...
TRANSCRIPT
![Page 1: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/1.jpg)
DAMS(a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”)
Kate WebbinkSharon Grant
Janeen JonesPete Herbst
Rob ZschernitzNorth American User Conference
2019
![Page 2: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/2.jpg)
[EMu Users 2019]
the silence of the DAMS
![Page 3: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/3.jpg)
A brief history of the FMNH EMu/DAMu-verse
Digitization
Projects
FMNH + EMu
the Future...
1,500,000
2,500,000
MM opened to
all users
2008
~
Catalogue
Multimedia
DAMS…
Standardization
Project…
2010
~
2012
~
2014
~
2016
~
2018
~
2015
~
2013
~~~ ~
2019
~
![Page 4: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/4.jpg)
A brief history of the FMNH EMu/DAMu-verse
Digitization
Projects
FMNH + EMu
the Future...
1,500,000
2,500,000
MM opened to
all users
2008
~
Catalogue
Multimedia
DAMS…
Standardization
Project…
2010
~
2012
~
2014
~
2016
~
2018
~
2015
~
2013
~~~ ~
2019
~
20 TB
$85 million
![Page 5: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/5.jpg)
DAMu usage timelines
![Page 6: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/6.jpg)
2008 - 2018...taking stock of ~10 year’s worth of Multimedia:
- File formats… not so standard- e.g., XLSX, 32 flavors of TIFF
- Filenames… also not so standard- e.g.: “DÈRP’s b.123(4).jpg”
- Files themselves… - sometimes go missing between EMu & storage → [broken icon]- dupdupduplicates
- Keywords… how to standard?- there was yelling...
[the screaming of the DAMS]
![Page 7: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/7.jpg)
...not so standard...
...some proprietary…
...reflect a user’s local registry...
File formats
![Page 8: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/8.jpg)
...not so standard...
...some proprietary…
...reflects user’s local registry...
File formatsDocumented standards…
Not set in stone, but reference-able
![Page 9: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/9.jpg)
...also not so standard
- “DÈ[R]P’s b.123 (#4).jpg”
= an example of a file that will disappear itself from a Multimedia export.
- ...mostly from the “È” character with the diacritic, - but if we’re fixing that, may as well pull out other special characters)
Filenames
ÁàÀâÂäÄãÃåÅæÆçÇéÉèÈêÊëËíÍìÌîÎïÏñÑóÓòÒôÔöÖõÕøØœŒßúÚùÙûÛüÜ\]\*'
![Page 10: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/10.jpg)
-----------sh*tty-filename-query:
...MulIdentifier contains '\*\[\-\\ ()~!@#$%^&*`,[]{}<>/?\;\:\'"\]\*'orMulIdentifier contains '\*\.\*\.\*'or MulIdentifier contains '\==\*\[áÁàÀâÂäÄãÃåÅæÆçÇéÉèÈêÊëËíÍìÌîÎïÏñÑóÓòÒôÔöÖõÕøØœŒßúÚùÙûÛüÜ\]\*'orSupIdentifier contains '\*\[\-\\ ()~!@#$%^&*`,[]{}<>/?\;\:\'"\]\*'orSupIdentifier contains '\*\.\*\.\*'orSupIdentifier contains '\==\*\[áÁàÀâÂäÄãÃåÅæÆçÇéÉèÈêÊëËíÍìÌîÎïÏñÑóÓòÒôÔöÖõÕøØœŒßúÚùÙûÛüÜ\]\*'...
Filenames
![Page 11: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/11.jpg)
Files themselves - when they’re over 2GB…
![Page 12: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/12.jpg)
Files themselves - when they’re over 2GB…get a little mangled
![Page 13: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/13.jpg)
...into the mystery-void between EMu & storage → [broken icon]
Files themselves - sometimes go missing
![Page 14: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/14.jpg)
/emufmnh/multimedia//1183/101//1183/102//1183/103/ (...!?)/1183/105//1183/106/
...into the mystery-void between EMu & storage → [broken icon]
Files themselves - sometimes go missing
- On each side of the void is an audit trail, (EMu audit + Filer log)
- but without cross-checking between systems…
EMu Filer
(the
void
)
![Page 15: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/15.jpg)
...go missing
...into the mystery-void between EMu & storage → [broken icon]
- On each side of the void is an audit trail, (EMu audit + Filer log)- but without cross-checking between systems…
Files themselves...
github.com / fieldmuseum / Collections-Scripts
- On each side of the void is an audit trail(EMu audit + Filer log)
- but without cross-checking between systems…
![Page 16: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/16.jpg)
...get re-re-dup-duplicated
- sometimes folks don’t look before they leap- Or they do, but their workflows get in the way
- (e.g., checking for dups by name...after renaming dups)
Files themselves...
![Page 17: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/17.jpg)
Out-of-sync duplicated data
Keywords
- Taxon Coverage: Text field, [S], read-through, limit = 500 characters
- Read-through of ObjObjectsRef_tab.DarScientificName
- Logic: Pipe delimited if Multimedia is attached to multiple Catalog records, and remove duplicates
- Vernacular Name: Text field, [S], read-through, limit = 500 characters
- Read-through of ObjObjectsRef_tab.IdeTaxonRef_tab.ComName_tab(1), where
ObjObjectsRef_tab.IdeFiledAs_tab = Yes
- Logic: Pipe delimit if multiple values are returned, and remove duplicates
- Related Geography: Text field, [S], read-through, limit = 500 characters
- Read-through of ObjObjectsRef_tab.DarCountry, DarStateProvince, DarCounty, DarOceanContinent
- Logic: Pipe delimit if MM attached to multiple Catalog records, and remove duplicates
- Temporal Coverage: Text field, [S], read-through, limit = 500 characters
- Read-through of ObjObjectsRef_tab.PalStrStratigraphyRef_tab.SummaryData
- Logic: Pipe delimit if multiple values are returned, and remove duplicates
- Sex: Text field, [S], read-through, limit = 100 characters
- Logic: Read-through of ObjObjectsRef_tab(1).DarSex
- Logic: Pipe delimit if MM attached to multiple Catalog records, and remove duplicates
- Life Stage: Text field, [S], read-through, limit = 100 characters
- Read-through of ObjObjectsRef_tab(1).DarLifeStage
- Logic: Pipe delimit if MM attached to multiple Catalog records, and remove duplicates
- Subject Orientation: Lookup List Text field (100 Chars), [S] ,
- Lookup list: NEW Subject Orientation Level 1
- Subject Part: Lookup List Text field (100 chars), [M]
- Lookup list: NEW, “Parts” - Level 2
- Capture Device: Lookup List Text field (255 chars), [S]
- Lookup list: NEW Capture Device - Level 1
- Access URI
- Logic: if AdmPublishWebNoPassword = Yes, and if MulDocumentType = “M” (not U or R), then
Concatenate “http://fm-digital-assets.fieldmuseum.org/” + [Multimedia] -- e.g.,
“http://fm-digital-assets.fieldmuseum.org/1801/052/UC_22544_single.jpg”
- Functionality: Display web link icon
- Related Resource ID: Text field, [S] , lookup list, limit = 100 characters
- Logic: If AssAssociatedRef_tab is not null, pipe-delimited AssAssociatedRef_tab.AdmGlobalUniqueID when
AssAssociatedRef_tab.AdmPublishWebNoPassword = Yes
- Derived From: Text field, [S], read-through, limit = 100 characters
- Logic: If RelParentRef is not null display RelParentMediaRef.[new Multimedia Identifier field] when
RelParentRef.AdmPublishWebNoPassword = Yes
- Funding Attribution: Text field, [S], read-through, limit = 100 characters
- Where DetContributorRole_tab = “Funder” and DetContributorRole_tab.AdmPublishWebNoPassword = Yes, pipe
delimited list of DetContributorRef_tab.SummaryData
- Identifier- AdmGlobalUniqueIdentifier with AdmGUIDPreferred = Yes
- Associated Specimen
- Logic: If ObjObjectsRef_tab.ColCollectionEventRef.irn is not null, show pipe delimited list of
ObjObjectsRef_tab.DarGlobalUniqueIdentifier where ObjObjectsRef_tab.AdmPublishWebNoPassword = Yes and
ObjObjectsRef_tab.CatSightingsEventsRef_tab.irn is NULL
- Associated Observation
- Logic: If ObjObjectsRef_tab.ColSightingsEventRef_tab.irn is not null, show pipe delimited list of
ObjObjectsRef_tab.DarGlobalUniqueIdentifier where ObjObjectsRef_tab.AdmPublishWebNoPassword = Yes and
ObjObjectsRef_tab.CatSightingsEventsRef_tab.irn is NOT NULL
-Taxon Coverage: Text field, [S], read-through, limit = 500 characters
-Read-through of ObjObjectsRef_tab.DarScientificName
-Logic: Pipe delimited if Multimedia is attached to multiple Catalog records, and remove duplicates
-Vernacular Name: Text field, [S], read-through, limit = 500 characters
-Read-through of ObjObjectsRef_tab.IdeTaxonRef_tab.ComName_tab(1), where
ObjObjectsRef_tab.IdeFiledAs_tab = Yes
-Logic: Pipe delimit if m
ultiple values are returned, and remove duplicates
-Related Geography: Text field, [S], read-through, limit = 500 characters
-Read-through of ObjObjectsRef_tab.DarCountry, D
arStateProvince, DarCounty, DarOceanContinent
-Logic: Pipe delimit if M
M attached to multiple Catalog records, and remove duplicates
-Temporal Coverage: Text field, [S], read-through, limit = 500 characters
-Read-through of ObjObjectsRef_tab.PalStrStratigraphyRef_tab.SummaryData
-Logic: Pipe delimit if m
ultiple values are returned, and remove duplicates
-Sex: Text field, [S], read-through, limit = 100 characters
-Logic: Read-through of ObjObjectsRef_tab(1).DarSex
-Logic: Pipe delimit if M
M attached to multiple Catalog records, and remove duplicates
-Life Stage: Text field, [S], read-through, limit = 100 characters
-Read-through of ObjObjectsRef_tab(1).DarLifeStage
-Logic: Pipe delimit if M
M attached to multiple Catalog records, and remove duplicates
-Subject Orientation: Lookup List Text field (100 Chars), [S] ,
-Lookup list: NEW Subject Orientation Level 1
-Subject Part: Lookup List Text field (100 chars), [M]
-Lookup list: NEW, “Parts” - Level 2
-Capture Device: Lookup List Text field (255 chars), [S]
-Lookup list: NEW Capture Device - Level 1
-Access URI
-Logic: if AdmPublishWebNoPassword = Yes, and if MulDocumentType = “M” (not U or R), then
Concatenate “http://fm-digital-assets.fieldmuseum.org/” + [Multimedia] -- e.g.,
“http://fm-digital-assets.fieldmuseum.org/1801/052/UC_22544_single.jpg”
-Functionality: Display web link icon
-Related Resource ID: Text field, [S] , lookup list, limit = 100 characters
-Logic: If AssAssociatedRef_tab is not null, pipe-delimited AssAssociatedRef_tab.AdmGlobalUniqueID when
AssAssociatedRef_tab.AdmPublishWebNoPassword = Yes
-Derived From: Text field, [S], read-through, limit = 100 characters
-Logic: If RelParentRef is not null display RelParentMediaRef.[new Multimedia Identifier field] when
RelParentRef.AdmPublishWebNoPassword = Yes
-Funding Attribution: Text field, [S], read-through, limit = 100 characters
-Where DetContributorRole_tab = “Funder” and DetContributorRole_tab.AdmPublishWebNoPassword = Yes, pipe
delimited list of DetContributorRef_tab.SummaryData
-Identifier
-AdmGlobalUniqueIdentifier with AdmGUIDPreferred = Yes
-Associated Specimen
-Logic: If O
bjObjectsRef_tab.ColCollectionEventRef.irn is not null, show pipe delimited list of
ObjObjectsRef_tab.DarGlobalUniqueIdentifier where ObjObjectsRef_tab.AdmPublishWebNoPassword = Yes and
ObjObjectsRef_tab.CatSightingsEventsRef_tab.irn is NULL
-Associated Observation
-Logic: If O
bjObjectsRef_tab.ColSightingsEventRef_tab.irn is not null, show pipe delimited list of
ObjObjectsRef_tab.DarGlobalUniqueIdentifier where ObjObjectsRef_tab.AdmPublishWebNoPassword = Yes and
ObjObjectsRef_tab.CatSightingsEventsRef_tab.irn is NOT NULL
![Page 18: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/18.jpg)
Out-of-sync duplicated dataMapping reverse-attached fields to Audubon Core:
Keywords Audubon Core
- Taxon Coverage- Vernacular Name- Related Geography- Temporal Coverage- Sex- Life Stage- Subject Orientation- Subject Part- Capture Device- Access URI- Related Resource ID- Derived From- Funding Attribution- Identifier- Associated Specimen- Associated Observation
![Page 19: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/19.jpg)
2018 - 2019...taking stock of 5 year’s worth of Multimedia + 1 year of cleanup:
- File formats… “stabilized”?- XLSX → open-docs formats; still need conversion tools for DNGs
- Filenames… renamed- “DÈRP’s b.123(4).jpg” → sh*ty-file-renamer.pl → “DRPs_b_123_4_.jpg”
- Files themselves…- sometimes go missing → check/compare logs between EMu & filer- Duplicates → backlog deleted + uniqueness on ChaMD5sum
- Keywords… mapped Audubon Core reverse attachments- there is less yelling...for now...
[the silence of the DAMS?]
![Page 20: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/20.jpg)
2019 - Setting up the DAMS server...So far, we’ve:
- set up the DAMS server (“Palmer”)- pointed it at our EMu server (“Shackleton)- [...testing stuff… (users / eevents / photo archives workflow?) ]
![Page 21: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/21.jpg)
All clear going forward?Or just the calm before the next storm?
(And who exactly is ‘Hannibal Lecter’ in this scenario?)
(And how many different misspellings of “DAMS” will there be?)
DD
![Page 22: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/22.jpg)
Citations/Sources- Clarice EMu / JodiEMu Foster - https://animals.sandiegozoo.org/animals/emu- Nathaniel B Palmer - - Ernest J Shackleton - - Shackleton crew with puppies - - Silence of the Lambs poster - © Orion Pictures - Screaming Lambs - https://twitter.com/screaminglxmb - Screaming Charlton Hestons - Soylent Green - © Metro-Goldwyn-Mayer- Death’s Head Moth - http://twmoth.tesri.gov.tw/peo/MothInfo/V19-20150515-174
- (Acherontia lachesis - occurrenceID: V19-20150515-174)
![Page 23: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/23.jpg)
Questions, Comments, Thought?(No real lambs were harmed in the making of this talk.)
Kate WebbinkSharon Grant
Janeen JonesPete Herbst
Rob ZschernitzNorth American User Conference
2019
![Page 24: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/24.jpg)
the silence of the DAMS
Jodie Foster = EMu?
![Page 25: (a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM](https://reader033.vdocuments.site/reader033/viewer/2022042401/5f0fbeb87e708231d445ac89/html5/thumbnails/25.jpg)
DAMu file types...