Community as Opportunity Launches a Study on File Formats
We have been digitizing documents in National Archives of Finland for almost 15 years and our capturing processes have developed a lot during that time. Our processes have developed; the amount of digital images has grown enormously especially during the last couple of years. Concerning digital images we have been using widely used formats to store and share our documents – such as TIFF (preservation) and JPEG (usage).
At the moment our digital images are basically copies from our analog world but after we put them through character recognizing processes they will “born again” and create completely different platform for the same information. After processing we need to know how, what kind and in which format we save our new manifestation since there seems to be no TIFF (standard) available for this kind of data!
Originally our main target in CO:OP in 2016 was to gather images for our first Topotheques. During the spring, however, we also started planning our future agenda in developing new tools that would make digitized cultural heritage more accessible and easier to use. In that respect a work group was gathered to make suggestions on what measures would benefit not only the National Archives of Finland but also other project partners.
During first meetings several issues were raised. They were about increasing service level for the digitized material, building tools for open data publication and management and advancing the use of persistent identifiers (PID), to mention some. In the end the group decided, that we should start by carrying out a study on file formats, especially as the National Archives is also partner in READ (https://read.transkribus.eu/). That was estimated to be a goal that would also benefit other memory and cultural heritage organizations not only in Finland, but also elsewhere. The idea of format study was introduced to CO:OP project management and it was approved without delay.
Building Up the Study
The objectives of the format study carried out have been twofold: to find out and describe existing file formats suitable for storing text recognition results and to analyze potential connections between the file formats (or their introduction) and other systems.
First, a list of file formats for further study was created and it consisted of e.g. PAGE XML, ALTO XML, ABBYY FineReader XML, hOCR, TEI and PDF. Then, information concerning both the historical/administrative and technical aspects of these formats was collected. Also, an attempt was made to discover already existing reports and recommendations addressing the listed file formats. The description of the technical aspects of the file formats dealt with e.g. how text and text areas are represented in the formats, how text recognition confidence is expressed and whether Named Entities (e.g. names of people, names of places) can be indicated. Names of specific tags and attributes to express those technical aspects were listed based on the existing documentation of the file formats and possible schema files.
Then, potential connections between the file formats and other systems were analyzed. This part consisted of the description of e.g. search system, system for metadata maintenance and system for long-term preservation and potential link that the introduction of the file format could have with them. For example, in case of search system, the introduction of a file format containing the recognized full-text of the archival document could utilize the whole textual content of the document, but it would at the same time tackle somehow issues such as text recognition errors and additional facets for search.
The report produced as a result of this file format study contains distilled information on the file formats and their potential implications, but also many references to other reports and publications that can provide further details and information on the issues addressed.
From collection management point of view this report is very interesting since our knowledge in this type of formats wasn’t that good. Now our knowledge has a grown a lot and we can base our decisions on facts that this report provides. Since we feel that it is important to share the report with all interested parties we decided to publish it under open Creative Commons license (CC BY). Therefore it is now available at http://home.icar-us.eu/documents/documents/2016/12/report-on-file-formats-for-hand-written-text-recognition-htr-material.pdf.