File formats as Emoji: 0xffae
by @beet_keeper
tldr: https://emoji.exponentialdecay.co.uk
File Formats As Emoji (0xFFAE or 0xffae) might be my most random file format hack yet. Indeed, it is a random page generator! But it generates random pages of file formats represented as Emoji.
The idea came in 2016 with radare releasing a new version that supported an emoji hexdump. I wondered whether I could do something fun combining file
@wyatt @kawa Until archive format overhead becomes the limiting factor for size.
At one point in early 2002, I briefly considered using the UNIX tape archive (tar) format to bundle assets in a Game Boy Advance game. Many of these assets were 2048 bytes or smaller. I looked up the spec for a POSIX tar file, and it involved rounding up each file's size to a multiple of the 512-byte block size and adding a 512-byte block header. That kind of overhead adds up. On top of that, searching for a particular file in a tarball takes linear time, not constant or even logarithmic time.
That led me to devise and document a simpler, more fit for purpose archive format called GBFS. Other specialized archive formats may benefit from packing files so as to avoid crossing 16K, 32K, 64K, or 128K block boundaries in the medium.
File format building blocks: primitives in digital preservation
by @beet_keeper
A primitive in software development can be described as:
a fundamental data type or code that can be used to build more complex software programs or interfaces.
– via https://www.capterra.com/glossary/primitive/ (also Wiki: language primitives)
Like bricks and mortar in the building industry, or oil and acrylic for a painter, a primitive helps a software developer to create increasingly more complex software, from your shell scripts, to entire digital preservation systems.
Primitives also help us to create file formats, as we’ve seen with the Eyeglass example I have presented previously, the file format is at its most fundamental level a representation of a data structure as a binary stream, that can be read out of the data structure onto disk, and likewise from disk to a data structure from code.
For the file format developer we have at our disposal all of the primitives that the software developer has, and like them, we also have “file formats” (as we tend to understand them in digital preservation terms) that serve as our primitives as well.
Huzzah! A small personal 15 year old goal achieved -- finally got a service to update automatically from the PRONOM release notes we first minted at TNA back in 2010 with signature file V31. (Rather than something going wrong and my needing to handle turn. We only trigger a download and update on a new release and it finally worked!)
Today's release V120 reflected live on the ffdev.info PRONOM dashboard and API.
The latest version of #PRONOM, v120 has been released! Identification for 30 new PUIDs, 32 new signatures and 32 updates. #digipres #fileformats #digitalpreservation https://www.nationalarchives.gov.uk/aboutapps/pronom/release-notes.xml
The sensitivity index: Corrupting Y2K
by @beet_keeper
In December I asked “What will you bitflip today?” Not long after, Johan’s (@bitsgalore) Digtial Dark Age Crew released its long lost hidden single Y2K — well, I couldn’t resist corrupting it.
Fixity is an interesting property enabled by digital technologies. Checksums allow us to demonstrate mathematically that a file has not been changed. An often cited definition of fixity is:
Fixity, in the preservation sense, means the assurance that a digital file has remained unchanged, i.e. fixed — Bailey (2014)
It’s very much linked to the concept of integrity. A UNESCO definition of which:
The state of being whole, uncorrupted and free of unauthorized and undocumented changes.
Integrity is massively important at this time in history. It gives us the guarantees we need that digital objects we work with aren’t harboring their own sinister secrets in the form of malware and other potentially damaging payloads.
These values are contingent on bit-level preservation, the field of digital preservation largely assumes this; that we will be able to look after our content without losing information. As feasible as this may be these days, what happens if we lose some information? Where does authenticity come into play?
Through corrupting Y2K, I took time to reflect on integrity versus authenticity, as well as create some interesting glitched outputs. I also uncovered what may be the first audio that reveals what the Millennium Bug itself may have sounded like! Keen to hear it? Read on to find out more.
Continue reading “The sensitivity index: Corrupting Y2K”…
The things that get you excited when you’ve spent too long looking at hex editors… receiving mojibaked invitations for the building’s Whatsapp group! (Although the text is strangely coherent when translated… so it could also be some random crossed wire from the provider)
Looking for some more advanced techies to help me out here. I was browsing the files of old abandonware (as one does) and came across the .zym file format in a game called Gubble 2. Does anyone have any idea what this file format is? Is it something proprietary by the gubble devs? Something that just isn't used anymore? The only thing google brought up regarding .zym was some mods for quake.
SUPER excited to announce a new fdd, FIRST ONE ever from Liz Caringola, on PAR (Parity Volume Set File Format Family - fdd634). We/Liz even submitted for the #PUID in #PRONOM and we'll do this as part of our workflow from now on. This format was a popular discussion topic with lots of community input. Comments always welcome but mostly if they say Go Liz! Great job! Woot! https://www.loc.gov/preservation/digital/formats/fdd/fdd000634.shtml #digipres #fileformats
I am trying to make a tool (#DJJerry) that helps you manage your local music collection. I've been stuck on trying to pick a format for the configuration/manifest files.
I have even thought about inventing my own file format designed to be very clean and readable to people who don't know code (shown in image).
What file format would you personally prefer?
Anyone else seeing the Just Solve It Wiki error?
What will you bitflip today?
by @beet_keeper
I want to let you into a secret: I enjoy corruption. Corrupting digital objects leads to undefined behavior (C++’s definition is fun). And flipping bits in objects can tell us something both about the fragility, and robustness of our digital files and the applications that work with them.
I had a pull-request for bitflip accepted the other day. Bitflip is by Antoine Grondin and is a simple utility for flipping bits in digital files. I wrote in my COPTR entry for it that it reminds me of shotGun by Manfred Thaller. The utility is exceptionally easy to use (and of course update and maintain written in Golang) and has some nice features for flipping individual bits or a uniform percentage of bits across a digital file.
My pull-request was a simple one updating Goreleaser and its GitHub workflow to provide binaries for Windows and FreeBSD. I only needed to use Windows for a short amount of time thankfully, but it’s an environment I believe is prevalent for a lot of digital preservationists in corporate IT environments.
Bitflip is a useful utility to improve your testing of digital preservation systems, or simply for outreach, but let’s have a quick look at it in action.
A year in file formats 2024
by @beet_keeper
A great write up from Francesca at TNA about the past year for PRONOM via Georgia and the OPF: https://digipres.club/@Georgia/113633592204634150.
It’s great to see the continuing work including vital translation of guides into other languages. Francesca includes a couple of shout outs to some pieces I have contributed in my spare time this year; including a collaborative workshop with Francesca, David, and Tyler at iPRES2024.
Our semi-annual blog post is out to recap recent work with #fileformats and #digitalpreservation at the Library of Congress. Highlights incl new format research on #finale music notation formats, participation in #iPres2024 and @anj's Registries of Practice project.
w/ Elizabeth M. Caringola, Genevieve Havemeyer - King, Liz Holdzkom and Marcus Nappier
https://blogs.loc.gov/thesignal/2024/12/file-format-research-roundup/
PRONOM’s dustiest records
NB. because of the complexity of this post, it may be easier to read in original blog form, than on Mastodon here: https://exponentialdecay.co.uk/blog/pronoms-dustiest-records/
Tyler’s recent blog post for the PRONOM Hack-a-thon Week 2024 (my previous for this week), brought up an interesting point about two of PRONOM’s oldest outline records, Real Video Clip (fmt/204) and Real Video (x-fmt/277). How did they end up in PRONOM?
Tyler suggests:
I assume PRONOM originally added these based on MIME types available.
I thought I knew the answer, and so it prompted a forensic look at the records to see if what I thought I knew aligned with reality!
As a PRONOM maintainer at The National Archives, UK from 2009-2012 I knew a little bit of the history of the system, we see some of that history impact us today, for example, when we look at the number of records that don’t have descriptions or file format signatures, 156 of those records are so-called x-PUIDs. A mechanism in PRONOM that was never meant to make it into the wild for working on file formats internally without polluting the public record. There are 455 x-PUIDs in total. They made it into the wild anyway (before my time) and so they exist as a symbol of PRONOM’s dustiest oldest records.
Even by the time I had started, PRONOM still had a lot of what we started to call outline records. One of the more positive changes we made to the process back in the day was that we would stop creating outline records; instead, we would focus on records that could be tied to signatures. This didn’t necessarily make the records more correctly aligned with reality, but it meant records had utility and file formats identified by DROID could be tied back to something that PRONOM “knew about”. I believe the process is a bit more flexible these days, allowing individuals to contribute information to records that tie them back to information like MIMEtypes and specifications. It’s clearer the format is “real” even if a signature is yet to be developed (and of course there are a large number of data formats that are hard to even represent in traditional PRONOM signatures any more and so they need a record, even if there isn’t a neat concept of a signature for them).
Okay old-man, but what about Tyler’s thesis?
Stellent and PRONOM
I learned sometime in my tenure at The National Archives that PRONOM had been seeded with a lot of the formats listed in a technology called OutsideIn previously owned by Stellent and now owned by Oracle.
Oracle OutsideInhttps://docs.oracle.com/outsidein/853/oit/OutsideIn (2010)https://web.archive.org/web/20101016164937/http://www.oracle.com/technetwork/middleware/content-management/oit-all-085236.htmlData sheet – Formats (2011)https://web.archive.org/web/20110125024733/http://www.oracle.com/technetwork/middleware/content-management/ds-oitfiles-133032.pdfCOPTR entryhttps://coptr.digipres.org/index.php/Oracle_Outside_In_TechnologyI had always had a feeling that that the scope of this list was largely exaggerated by the company selling the software as it is a marketing tool; and if not exaggerated, perhaps, just not as clearly delineated by format than PRONOM, and rather, by Software, regardless of the properties of a given “format”, e.g. WinZip, and PKZip.
Back to the story though, I was also reasonably sure I would find Tyler’s RealVideo formats in the format listing but, I did not!
I downloaded a CSV summarizing the PRONOM records from api.pronom.ffdev.info with:
curl -X 'GET' \ 'https://api.pronom.ffdev.info/pronom_summary_csv' \ -H 'accept: application/csv'
I filtered on outline entries and those without signatures only. I went through the entries still remaining and looked for name matches. I did find some name-for-name matches and some that were close, but no RealVideo or RealVideo Clip.
The matches:
7-bit ANSI Textyes7-bit ASCII Textyes8-bit ANSI Textyes8-bit ASCII TextyesEBCDIC-USyesFramework Database IIIyesIBM DisplayWrite Document 2yesIBM DisplayWrite Document 3yesMicrografx Designer 3.1yesNota Bene Text FileyesUnicode Text FileyesThe maybes:
Cascading Style SheetmaybeFreelance File 1.0-2.1maybeMacPaint GraphicsmaybeMicrosoft Office Binder File for Windows 95maybeMicrosoft Works DatabasemaybeMicrosoft Works Database for DOS 2.0maybeMicrosoft Works Database for Windows 3.0maybeMicrosoft Works Database for Windows 4.0maybeProfessional Write Text FilemaybeWordPerfect for Windows Document 5.2maybeXYWrite DocumentmaybeXYWrite Document IIImaybeXYWrite Document III+maybe11 exact matches! It’s hardly a headline!
I had hoped that if I found more exact matches it would provide some clues to where some of the older PRONOM entries came from. I expected most of the outline records to come from this list, alas, it isn’t nearly as many as anticipated.
I hoped too that going through the list I might get more clues as to formats that could potentially be deprecated in PRONOM.
As it stands, from the OutsideIn list, the only records I would personally recommend for deprecation are:
7-bit ANSI Text7-bit ASCII Text8-bit ANSI Text8-bit ASCII TextEBCDIC-USUnicode Text File
We know enough now to be almost certain that if something that looks like these files arrives in the archive it will present as a standard text file, and that we will need to rely on determining the character encoding using tools such as Richard Lehane’s characterize (see characterize’s README for more background). It is unlikely we will be able to attach a signature to these records, and we know there are a great deal more encodings in the world than need be represented as PRONOM identifiers.
NB. this might be something to formalize in a PRONOM decision making rubric, connected also, to formalizing approaches for XML based signatures.
A bit of a let down, or is it?
Still uncomfortable with so many outline records and little provenance for them, I wanted to find more information about the source of PRONOM data and so I decided to take a different path — I surfed the internet for answers!
Out of the list of outline records I found a few to be overly specific, or slightly weird, i.e. not really things we hear much about day-to-day, some examples:
ACBM GraphicsApple SoundAutoCAD Plot Configuration File 1.0-R13AutoCAD Plot Configuration File R14AutoSketch DrawingBtrieve Database 5.1CorelDraw PatternDEC Data Exchange FileDEC WPS Plus DocumentDr Halo BitmapGeneric Library FileHTML Extension FileHewlett Packard AdvanceWrite Text FileInkwriter/Notetaker TemplateInset Systems BitmapInstalit ScriptInterleaf DocumentMicrosoft Excel Add-InMicrosoft Excel ODBC QueryMicrosoft Excel ToolbarMicrosoft Powerpoint Design TemplateMicrosoft Print FileMicrostation CAD Drawing 95NAP MetafileNota Bene Text FileOS/2 Change Control FileRevit External GroupSAP DocumentSAS Data FileScanstudio 16-Colour BitmapSchedule+ ContactsSpeller Custom DictionaryUnisys (Sperry) System Data FileWordperfect Secondary File 5.0Wordperfect Secondary File 5.1/5.2form*Z Project FileACBM graphics? Dr Halo Bitmap? Btrieve database, “5.1”? where are the other five?!!
It gave me pause. I didn’t believe these were all formats well-known to folks who created PRONOM, and I know we didn’t have such an advanced digital transfer program at the time that meant agencies were submitting huge variations of formats to PRONOM for future preservation.
I felt they had to come from somewhere, but where?
Enter Filext.com
Because these formats were very specific I found listings on the internet that I knew had to be part of the story. I had immediate luck just looking for combinations of these names, e.g. ACBM Graphics + NAP Metafile.
In particular I found listings on different websites from hobbyists or universities that all looked the same or similar, e.g.
There were definite matches with PRONOM which we will get to, but I started to wonder about the provenance of these extensions.
I kept looking and I found one clue, a header and footer of a file that looked like those above and read as follows:
Copyright © 2002 Computer KnowledgeAll Rights ReservedThis download for personal use only. Do NOT distributeit to others either alone or incorporated into anysoftware without prior permission from Computer Knowledge.Developers who wish to incorporate portions of the listplease see the comments at the end of this file.
Developer permissions....This total file may not be included in any other software orproject which presents the data to the public or portions ofthe public. Any developer who wishes to include up to (butnot more than) 2,000 individual entries from this file is freeto do so provided certain conditions are met. These are:. 1) Credit must be given to FILExt. If links are available in the developed product then one must also be provided to FILExt as http://filext.com.. Suggested text: "File extension list courtesy of FILExt. For a more extensive list visit http://filext.com.". 2) Once the extensions are chosen for one product by any developer then these same extensions must continue to be used by that developer for any other projects (i.e., you cannot take one set of 2,000 for one project and a different set of 2,000 for another project; it's a total of 2,000).. 3) If links are available in the developed product then any links appearing associated with any of the 2,000 picked extensions must be included in the product. (This covers future plans to include such links in this list.).When the project is complete please notify FILExt with thespecifics at feedback@filext.com. We're always interestedin how the list is being used. Thank you.
Filext.com!
And so I asked myself, how long had filext been around?
As it turns out, quite a while! It was forked from a site called cknow around 2002. cknow.com was registered around 1996 and filext.com registered in 2001.
The first appearance of cknow in the internet archive is late 1996: https://web.archive.org/web/19961219035827/http://www.cknow.com/ and Filext early 2001: https://web.archive.org/web/20010522235126/http://www.filext.com/
The sites were founded by Tom Simondi. It looks like he has been responsible for a lot of the 90s and 00s work around demystifying extensions and getting more information to folk about what to do with them.
Could it be the source of the first PRONOM records?
Comparing some of the many other text-based lists I had found with cknow and filext gave me some confidence that there was some shared heritage with the them, and so I asked, could the cknow and filext lists have also seeded PRONOM?
I picked a list close to 2002 (cknow Extensions: 2000) when PRONOM was first started and began to compare entries for exact matches.
ACBM GraphicsyesAutoCAD Compiled MenuyesAutoSketch DrawingyesBtrieve Database 5.1yesDataFlex Query Tag NameyesDeluxe Paint bitmapyesDesignCAD DrawingyesDigital VideoyesDr Halo BitmapyesFrame Vector MetafileyesFramework Database IIyesFramework Database IIIyesFramework Database IVyesInformation or Setup FileyesInset Systems BitmapyesInterBase DatabaseyesLotus Approach View FileyesMathematica NotebookyesMicrosoft Excel Add-InyesMicrosoft Excel ODBC QueryyesMicrosoft Excel OLAP QueryyesMicrosoft Excel OLE DB QueryyesMicrosoft Excel Web QueryyesMicrosoft FoxPro LibraryyesMicrosoft Outlook Address BookyesMicrosoft PowerPoint Graphics FileyesMicrosoft Powerpoint Add-InyesMicrosoft Visual FoxPro TableyesMicrosoft Works DatabaseyesMicrosoft Works DocumentyesMicrostation CAD Drawing 95yesNAP MetafileyesNota Bene Text FileyesOS/2 Change Control FileyesPICS AnimationyesPageMaker Document 3.0yesPageMaker Time Stamp File 4.0yesProfessional Write Text FileyesQuicken Data FileyesRealVideo Clip <– cc. Tyler!yesSchedule+ ContactsyesStatGraphics Data FileyesStructured Query Language DatayesVentura Publisher Vector GraphicsyesXYWrite Document IIIyesXYWrite Document IVyes46 matches!
Apple SoundmaybeAutoCAD Device-Independent Binary Plotter FilemaybeAutoCAD Drawing TemplatemaybeCascading Style SheetmaybeDEC Data Exchange FilemaybeDEC WPS Plus DocumentmaybeFreelance File 1.0-2.1maybeJava Servlet PagemaybeMicrografx Designer 3.1maybeMicrosoft Office Binder File for Windows 95maybeMicrosoft Office Binder Template for Windows 95maybeMicrosoft Office Binder Template for Windows 97-2003maybeMicrosoft Office Binder Wizard for Windows 95maybeMicrosoft Office Binder Wizard for Windows 97-2003maybeVentura PublishermaybeXYWrite DocumentmaybeXYWrite Document III+maybe17 maybes!
What did we answer?
Okay, 46 exact matches does not the full listing make (although many (now) full-entries may still have been made from these early listings). Filext may have been an important resource for the first PRONOM records, but it’s also likely that PRONOM had other sources of information. For example, for a number of the Microsoft formats with outline records read like export or save-as listings in previous versions of Microsoft software. E.g. Excel:
NB. I wasn’t actively researching this side of things writing this blog, but I can already see some commonalities, especially Unicode Text!
I know we also had a copy of the Dr Dobb’s Essential Books on File Formats CD-ROM in the archive, and so that may also have been an important resource when PRONOM was creating its first records.
I count only two overlaps with the Stellent list, Framework Database III and Nota Bene Text File.
We did, however, find the RealVideo Clip! And I think we found some decent correlation with a resource that looks likely to have been used partially to populate the PRONOM database.
The era of file extensions
I am sure we looked at these sites during my time on PRONOM, although with less frequency given the need to reduce outline records and increase the number with actionable information.
NB. I also learned that TrID has been around since 2003! https://web.archive.org/web/20030612031252/http://mark0.ngi.it:80/
Provenance and prior art
It’s not entirely productive to say I wish we had better provenance for PRONOM records back in the day – but I do!
It makes me reflect on the importance of looking outside of our own walls in digital preservation instead of the constant redundancy of reinvention or ownership.
Often as academics, or those with archival views of the world, we can provide a polish and precision to technology as it exists to make it more usable in an archival context.
But cknow has been around so long, and the Unix utility File was created in 1986.
There’s a parallel history here that we should be recognizing and sharing for our next colleagues.
I arrived at TNA in 2009 and learned about File maybe two years later. As a Windows guy at the time, that might not be uncommon, but I do feel it is on me to have known more. I also think it should have been trivial to access the provenance around some of the records in the database at the time, but more than that – as a field, shouldn’t we all know Tom Simondi? What if the same academic rigour of PRONOM and DROID could have been applied to existing tools like File? What if we had expanded our bubble and recognized digital preservation (or the tools for it) is something people have been doing in all but name for the longest time? What if the people working in parallel on these projects and websites were part of the digital preservation inner-circle community today?
I don’t have answers, but I feel there are lessons there for the future. Not reinventing or rebuilding without good reason is important, but even if we build something new and we have been inspired by something else, continuing to recognize and acknowledge prior art is important.
What do you think?
Also, how do we get these people into a room and celebrate their work, and learn more!
What next?
I don’t think I got very far here but I found it interesting, and I hope other readers may as well.
This is meant to be a PRONOM hack-a-thon blog and I don’t know if I have pushed the sticks forward that much but maybe there’s a bit more to reason about in the outline records, for example, around the plain-text formats mentioned above and a few more identified along the way.
7-bit ANSI Textx-fmt/21Recommend deprecation7-bit ASCII Textx-fmt/22Recommend deprecation8-bit ANSI Textx-fmt/282Recommend deprecation8-bit ASCII Textx-fmt/283Recommend deprecationUnicode Text Filex-fmt/16Recommend deprecationEBCDIC-USfmt/159Recommend deprecationMS-DOS Text File with line breaksx-fmt/130Recommend deprecationI noticed in the outline entries some low-hanging fruit that I might focus on next opportunity if someone else doesn’t get there first, these would be:
Cascading Style Sheetx-fmt/224Consider adding CSS to the record nameA signature should be feasibleDocument Type Definitionx-fmt/315Consider adding DTD to the record nameA signature should be feasibleExtensible Stylesheet Languagex-fmt/281Consider adding XSL to the record nameA signature should be feasibleHTML Extension Filex-fmt/417Related to Microsoft’s ISS serverA signature may be possibleStandard Generalized Markup Languagex-fmt/195Consider adding SGML to the record nameA signature may be possibleStill Picture Interchange File Format 2.0fmt/113Related to JPEGA signature should be possibleStructured Query Language Datafmt/206Consider adding SQL to the record nameA signature may be possibleDreamweaver Lock Filefmt/335A system file, there may be an entry in the NSRL databaseA signature may be possibleA little more on the history of extensions websites
The complete filext text file (allext.zip)
It took a few jumps, but I found the complete downloadable text file from Filext.com. I don’t think it exists any more and I don’t think the internet archive managed to grab a copy. Apparently it was quite a chunk of data to download on the web once upon a time, but they eventually found a way to release a zipped text file:
Via one jump we get to the “whole list” page:
https://web.archive.org/web/20020605164206/http://filext.com/wholelist.htm
And then to confirm our absolute interest in downloading it, we get to the a2z file:
https://web.archive.org/web/20020606071418/http://filext.com/a2z.htm
Which would have taken us to the zip file, alas, never captured on the Internet Archive anyway, maybe it is on other Memento compatible servers:
https://web.archive.org/web/20060117000000*/http://www.filext.com:80/allext.zip
Keeping filext up to date
Filext still asks for registry data to help keep it up to date. That’s pretty cool!
https://filext.com/faq/gather_data_for_filext.html
1 │ Echo OFF2 │ CLS3 │ assoc > filext_submission_output.txt4 │ Echo ---------- >> filext_submission_output.txt5 │ ftype >> filext_submission_output.txt6 │ Echo Thank you. The output file has been created and7 │ Echo named filext_submission_output.txt and it should8 │ Echo be in the same place where you saved this batch9 │ Echo file. All that is left now is to send that file10 │ Echo to FILExt. Attach it to an E-mail sent to the11 │ Echo address: filetype@filext.com12 │ Echo The E-mail subject should be: Submission13 │ Echo Thank you.14 │ Pause15 │ Exit
Filext as a source of learning
The filext faqs and community seemed particularly helpful and interesting back in the day:
https://web.archive.org/web/20090322040812/http://filext.com/faq/
File extension aggregator
The file-extension.net website started an aggregator project around 2007 and it’s still running today!
http://file-extension.net/seeker/
Some bonus images…
As I was working on this, I found irony in Google Sheets glitching, I managed to grab some screenshots along the way. Thanks for reading everyone!
“as a result of these choices, we now have three incompatible FASTQ variants that cannot be reliably distinguished.“
Huh, looking through some old links from 2015. Forgot about fileformat.info especially the screeds of updated articles on the front page.
literally who hurt genomics to make you all encode one specific kind of number as the ASCII characters from ! to ~ as an integer input to some logarithm function, but then others of you changed the function but kept encoding it as a single ASCII character ranging from -5 to 62 (???), and then later they decided that -5 to 62 was silly and so they changed that to 0-62 and throwing away half the original range for no reason, except actually it's 0-40 by convention.
did anyone consider "encoding it as a number"
simpledroid: completing the circle
It’s nearing the end of 2024 and that must mean a PRONOM hackathon as part of the World Digital Preservation Day (#WDPD2024).
My contribution is a follow-up on my work earlier in the year to produce a valid DROID signature file from Wikidata in wddroidy.
simpledroid is available on GitHub and creates a simple DROID signature file from PRONOM itself, creating a scripted pathway to create a signature file using official PRONOM data that doesn’t require the current PRONOM database and its legacy stored procedures.
It also does away with a lot of the excess data in the current DROID signature file which was previously an optimization for its Boyer Moore Horspool search algorithm, as described by Matthew Palmer.
The primary reason for simpledroid was to complete the circle on my previous efforts and to prove that it was possible to create a simplified signature file and for it to work with DROID. The result is about 80-90% there, with only a few skeleton files that remain unidentified – it should only require a small amount of forensic research to determine the reason.
The output provides a way for simplifying the signature file generation process, offering new opportunities to create alternative versions, or filtering what’s already there, e.g. filtering out any signatures that aren’t explicitly for image identification, e.g. in a digitization workflow.
It may provide another way into PRONOM data for those who might look at DROID first as well as opening up different ways to modify and test signatures.
It is possible to see in the reference output, that the signatures are much easier to understand via this simplified DROID file.
simpledroid outputs a file with a smaller footprint than the current file:
1.2M DROID_SignatureFile_Simple_2024-11-11T12-29-22Z.xml3.4M DROID_SignatureFile_V118.xml
It also contains all of the file classification data e.g. FormatType="Video" from PRONOM that will be added into DROID in a future release (and is already available in Siegfried).
Unlike the wddroidy work, priorities have also been added to the signature file so the mechanics of the signature file are pretty close to the official version (DROID uses the signature sequence and offsets to identify a file, but it then uses a priority to determine what results to display to the user where there may otherwise be positive matches for formats that provide the foundation for another, e.g. how XML forms the basis of SVG or XHTML.
It might be possible to remove some data around minimum and maximum offsets in the new file after discovering that simplified droid syntax requires curly bracket syntax at the beginning and end of sequences to mimic the same behavior, e.g.
With a BOFoffset, min_offset = 2, and signature = BADF00D1, the signature needs to become {2}BADF00D1 to work.
The code is pretty straightforward and uses a few tricks to output XML sensibly without having to build the document’s tree (DOM) in a more verbose way. There are probably a few other shortcuts I’d fix with time if the code was ever useful, including improving variable naming and adding tests.
I’m not sure this code will ever be needed, or used by anyone, but for a quick hack and a quick proof of concept, it felt good to put it out there. Maybe someone will look at this or the wddroidy work and see there may be a way to federate different sources of signature information together into something DROID can use. Or it might be a useful demonstration to the DROID team that allows them to simplify PRONOM’s database and output mechanisms in a way that remains compatible with existing tools.
Previous research week work
My previous work for PRONOM research week includes a dashboard and API for getting more information out of PRONOM, including listings of those records still requiring descriptions or signatures. You may find that work interesting and it is available at https://pronom.ffdev.info and https://api.pronom.ffdev.info.
And if you want to get in on the signature development work, signature development utility 2.0 (https://ffdev.info) was also a previous effort of mine for research week 2020 and will hopefully also benefit from outputting DROID’s simplified syntax.
A week of file formats
Of course with World Digital Preservation Day, file formats were pretty popular.
Andrew Jackson attempted to calculate how many distinct formats might be out there using methods used to calculate ecological diversity.
Amanda Tome described the scope of their work and shared a number of useful resources including useful links to the PRONOM starter pack and to the PRONOM drop-in sessions.
You might also find out a bit more about yourself by playing this File Format Dating Game from Lotte Wijsman and colleagues: Susanne van den Eijkel, Anton van Es, Elaine Murray, Francesca Mackenzie, Ellie O’Leary, and Sharon McMeekin. (I ended up on a date with FASTA (FDD000622) in my first play-through!)
Not specifically for WDPD, but in the same week I also enjoyed this presentation from Ange Albertini looking at different ways of identifying file formats. One big take away for me was thinking about how to get more forensic information out of a file format identification. DROID doesn’t tell us a lot, but is there a world in which one day it could?
Let me know if you find any of this work useful at all; and good luck on your file format endeavors this week.
️ #Vagabond 
