Originally called Media Market Technologies, MEMAT has grown from a picture library system into a fully-fledged Archival Digital Repository System applicable for art, heritage, scientific and media collections. After almost two decades of development, Africa Media Online has launched MEMAT 4 with an all-new web interface.
Understanding the Need
Any organisation wanting to preserve their identity needs a Digital Archive
Most organisations have a gap appearing in their history between the point at which their physical archive ends and the present, with born-digital files generated over the past decade left ungathered and unprocessed.
The gap is often populated by lost digital files because, for example, they were saved onto a stiffy disk that can no longer be read, or a DVD has become corrupted, a hard drive ‘ticks’ every time you plug it in, your computer contracted a virus, or you mistakenly neglected to rename files and so you overwrote older ones when you copied them to the same folder. Digital files are easy to create and even easier to lose!
Even if the born-digital files exist, they are often scattered across the organisation on various computers or hard drives and “we’re sure that so-and-so has them on a DVD somewhere” becomes a phrase that is all too familiar.
Even if files are gathered to a server somewhere, often they tend to be saved in a folder structure with folders and subfolders and sub-subfolders and frequently only one or two people know how to operate that structure, which means that if they leave the organisation no one can locate anything.
Even if files are on a server, usually that server can only be accessed by internal staff, and so your Digital Archive is not available to the whole community you serve.
Even if you were to make the files available to your community or target audience, you may fear what they will do with them and how you will manage access and use rights. Further you dread the time and energy that is likely to be required to service the masses of requests you will receive from your community.
The solution is an Archival Digital Repository System. All organisations have a Raw Digital Archive (the scatter of digital files across the organisations). Some have a Processed Digital Archive (the gathering of files in one place, usually a server being placed in some sort of order). Few organisations however select out the best of these and place them on a Public Digital Archive. A Digital Archive hosted on an Archival Digital Repository System is the permanent home for the Public Digital Archive and becomes the official record of your organisation.
MEMAT is made up of a number of layers
Building on the approach we took in MEMAT 3, MEMAT 4 is built using components that interface with one another through Application Programming Interfaces (APIs). Each component of the system is designed to do one thing well and interface with other self-contained components. This means that we can easily swap old components with newer ones using the latest technology. While such components occur at every level through the system, this approach has also been applied to the macro level of the system – what we call Layers.
The Infrastructure Layer is the hardware upon which MEMAT runs. This is made up of servers, backup hardware and the network between them. Africa Media Online’s specialised Infrastructure Layer is known as our ‘Preservation Cloud’.
On top of the Infrastructure Layer is the Operating System Layer. Like the operating system sitting on your computer, all programmes need to run on top of the operating system layer. In the case of MEMAT, this is a widely-used Open Source operating system – Linux, Apache, MySQL, PHP (LAMP).
The Asset Management Layer (AML) is the real power behind MEMAT. It is the part of the system that actually looks after the digital files and ensures they are backed up. It is made up of a Digital Vault which houses Security Boxes in which the digital files reside and it keeps track of all those files and their derivatives. The AML is developed using Perl, a powerful Open Source programming language.
The Cataloguing Layer is where information about the digital files is captured. This information is called metadata. MEMAT has a Metadata App that enables the capture of metadata in the fields of various open metadata schemata. The Metadata App has been developed using the Ruby on Rails Open Source programming language.
The Presentation Layer is the part of the Digital Repository that the users get to see and is the web interface for the MEMAT system. It has been developed in such a way to incorporate or interface with numerous Open Source systems. It is built on the Open Source CMS, WordPress, that uses the PHP programming language, the system also utilises IIIF technology.
Asset Management Layer: The AML
MEMAT’s AML preserves and secures the digital files
With over a decade of experience in working with media files at global standards, heading up the team that wrote the best practices section of a National Policy on the Digitisation of Heritage Resources, as well as experience in assisting heritage and scientific organisations to write strategies for digital archiving, Africa Media Online brings a wealth of knowledge in the production of media files at the right quality. Such standards are built into MEMAT and ensure long-term accessibility and usability of media files and their associated metadata.
MEMAT is a multimedia system preserving and giving access to digital image, manuscript, audio and video files. MEMAT’s Asset Management Layer supports all media types and common media file formats. Our emphasis is on open file formats that are extensively supported internationally to ensure conformity to long-term preservation requirements.
Digital files are ingested into the Digital Vault where they are stored permanently in DVD or BluRay size Security Boxes. This means that they can be copied onto BluRay disks for offline backup. Once a digital file is ingested into a Security Box, it is immutable. It can be superseded by a new version but it cannot be changed thereby preserving digital authenticity.
When digital files are ingested into the Digital Vault, derivative files used for access are autogenerated from the preservation file including thumbnails, previews and low-res files for image, manuscript, audio and video formats. This means that only the preservation quality file needs to be ingested and the system does the rest of the work.
When digital files are ingested into the Digital Vault they are checksummed so that the system can detect changes to the file over time and can be restored from backups should the file become corrupted.
A vital preservation service that MEMAT performs, in addition to the maintenance of your digital archive, is the backup of digital files. The files are stored in disk arrays that use Raid 5 and are also backed up onto LTO tapes and to BluRay disks for off-site storage. This ensures the migration of media files over time to new storage media.
The AML also stores all metadata relating to a particular file and makes that available to the search engine so that the PL can draw on the data to enable search and discovery of the file.
Cataloguing Layer: The Metadata App
MEMAT’s Metadata App enables the assigning, capture and quality control of metadata
The Administrator Login allows the Administrator to navigate the structure of the archive and identify and assign groups of material to appropriate Metadata Capturers. They also receive completed work back from those Capturers and can quality control that work, including correcting mistakes and providing feedback on the work done. When the Administrator approves the work, it is made live on the MEMAT PL where users are searching and browsing. Administrators can also create multiple capturer accounts on the system.
When Metadata Capturers log into the system, they can see the work assigned to them and are able to select items to work on and add metadata. The Metadata App utilises a number of metadata schemata including Dublin Core, IPTC and Darwin Core for science museums. A Capturer can chose to work on more than one item at a time if there are common elements that need to be added to all selected items. Once complete, committing the changes sends the work back to the Administrator for quality control.
Presentation Layer: The PL
MEMAT’s Presentation Layer is the web interface for the MEMAT system
The MEMAT 4 PL allows for different levels of password protected access. An unlogged-in user has access simply to search, browse and view galleries. In order to make selections and order files, however, the user must first register on the system and be logged in. An Administrator has access to the backend of the system and can manage users and their access, manage file orders, create galleries and edit metadata. Designers can also be given a login to make design changes to the look and feel of the web interface.
The MEMAT 4 PL has a Design Layer whereby the web interface can take on the unique look and feel of your organisation. It uses a fully responsive design that can work equally well on a large screen desktop or a screen of a smart phone.
The MEMAT 4 PL enables Administrators to create galleries or select materials for all users of the system to see. It also allows users to create private selections that they can share with others. Selections can only be seen by the logged-in user and others with whom the user chooses to share material.
Drawing on the search engine in the AML, the MEMAT 4 PL presents search results beautifully. For manuscripts it can draw on the Optical Character Recognition (OCR) text and on associated metadata enabling the system to return results that conform to the search term, and then, if logged-in as a registered user, by clicking on a particular result, it will search for that term within the manuscript. The system also allows users to browse the structure of the archive both in the browse tab and from the preview page of a particular search result.
MEMAT aims at providing both digital preservation and access. It manages the processes around the order and delivery of files. Users cannot gain access to preservation quality digital files unless they are logged in and order the file. MEMAT 4 puts the control for vetting orders and releasing files in the hands of the System Administrator.
Africa Media Online is the first organisation in Africa to adopt the International Image Interoperability Framework (IIIF) the new international standard for the presentation of heritage and scientific collections online. IIIF brings advanced technology that enables the rapid roll out of immersive online experiences by interacting with digitised collections, such as deep zoom of materials and searching within manuscripts.
Africa Media Online uses the open source WordPress Content Management System (CMS) as a base for the MEMAT PL. With 33% of all websites on the internet powered by the system, it provides a familiar backend for System Administrators enhancing MEMAT’s user-friendliness. While it has a reputation for security issues, at its core it is just as secure as Drupal and Joomla. Africa Media Online has built on that core, disallowing extraneous plugins. WordPress also gives us the ability to rapidly develop new features.
Infrastructure Layer: Preservation Cloud
MEMAT and the digital files it preserves are hosted on the Preservation Cloud
Cloud storage enables the distribution of media files across servers situated all over the World presenting both a benefit and a security challenge. When Africa Media Online set out to build the Preservation Cloud, we were aware of the need to construct a solution that benefits from cloud storage while at the same time minimises the security risk. Preservation files are therefore stored on servers in our buildings and only derivative files are made available through a web server that is located in South Africa’s jurisdiction. Now you can both know where your data is and access it in a secure way from anywhere in the World.
The Africa Media Online Server Room is located in a different building from our main offices where our Backup Room is sited. The Server Room is fitted with biometric access and monitoring, fire suppressant systems and other security measures. It has an inverter and batteries that provide 12 hours of backup power and a backup generator that can support the batteries when they are running low. The Server Room hosts the main storage servers, the internal web server and the processing server used to process material that is uploaded onto the MEMAT Digital Vault.
The Backup Room is located within Africa Media Online’s main offices. It is connected to the Server Room via fiber optic cable so that backups are automatically offsite. The Backup Room houses our LTO tape libraries and has the capability to back up to both LTO6 and LTO7 tapes. In addition we have a system to burn data onto BluRay disks providing another level of offline backup. This room is temperature and humidity controlled and has a backup power supply through an inverter and a generator as well as backup internet.
The LTO tapes and BluRay disks that have data written to them are stored in a safe within a fire-proof building in a rural area 20 km away from our main offices.
Derivative access files are synchronised to our external Web Server located in a secure data centre located in the Gauteng Province, South Africa. We are currently working on synchronising this to an internal web server with an automated failover to ensure uninterrupted service.