access | Spellbound Blog

Preserving Virtual Worlds – TinyMUD to SecondLife

August 17, 2007 3 Comments

A recent press release from the Library of Congress, Digital Preservation Program Makes Awards to Preserve American Creative Works, describes the newly funded project aimed at the preservation of ‘virtual worlds’:

The Preserving Virtual Worlds project will explore methods for preserving digital games and interactive fiction. Major activities will include developing basic standards for metadata and content representation and conducting a series of archiving case studies for early video games, electronic literature and Second Life, an interactive multiplayer game. Second Life content participants include Life to the Second Power, Democracy Island and the International Spaceflight Museum. Partners: University of Maryland, Stanford University, Rochester Institute of Technology and Linden Lab.

This has gotten a fair amount of coverage from the gaming and humanities sides of the world, but I learned about it via Professor Matthew Kirschenbaum‘s blog post Just Funded: Preserving Virtual Worlds.

The How They Got Game 2 post Library of Congress announces grants for preservation of digital games gives a more in depth summary of the Preserving Virtual Worlds project goals:

The main goal of the project is to help develop generalizable mechanisms and methods for preserving digital games and interactive fiction, and to begin to test these mechanism through the archiving of selected test cases. Key deliverables include the development of metadata schema and wrapper recommendations, and the long-term curation of archived cases.

I take this all a bit more personally than most might. I was a frequent denizen of an online virtual world known as TinyMUD (now usually referred to as TinyMUD Classic). TinyMUD was a text based, online, multi-player game that existed for seven months beginning in August of 1989. In practice it was sort of a cross between a chat room and a text based adventure. The players could build new parts of the MUD as they went – in many ways it was an early example of crowdsourcing. There was a passionate core of players who were constantly building new areas for others to explore and experience – not unlike what is currently the case in SecondLife. These types of text based games still exist – see MudMagic for listings.

Apparently August 20, 2007 will be TinyMUD’s 18th Annual Brigadoon Day. It will be celebrated by putting TinyMUD classic online for access. The page includes careful notes about finding and using a MUD Client to access TinyMUD. The existence of an ongoing MUD community of users has kept software like this alive and available almost 20 years later.

With projects like Preserving Virtual Worlds getting grants and gaining momentum it seems more plausible with each passing day that 18 years from now, parts of 2007’s SecondLife will still be available for people to experience. I am thankful to know that a copy of the TinyMUD world I helped build is still out there. I am even more thankful to know that the technology still exists to permit users to access it even if it is only once a year.

Update: 20th Anniversary of TinyMud Brigadoon day is set for Thursday, August 20, 2009

Public.Resource.Org: Creative Financing and Public Domain Content

July 29, 2007 1 Comment

Public.resource.org is dedicated to using funds contributed by individuals to buy public domain content. This content is then released online in multiple locations such as the Internet Archive and Google Video for use by anyone. I love their tag line: Underwritten By The Feds! Overwritten By You!

I spotted this in boingboing’s post Liberated public domain government docs surfacing online and I was immediately intrigued. This isn’t really an archiving issue exactly – though you could decide that it takes more of a LOCKSS approach to preservation. I also wonder how this approach could be used to finance the digitization of other public domain materials.

The website explains on their About Us page that they have recently applied for non-profit status with the IRS, so soon the purchase price of these materials could become a tax deduction for those who file US Tax Returns. They feature materials from 54 different US Federal agencies – from the Fish and Wildlife Service to the IRS. There are materials on the Environment, Public Health, Flying and many more.

But that isn’t all they are tackling – back in May they issued a message to The Internet discussing their attitude toward (and frustration with) the Smithsonian Images website. It begins:

We write to you today on the subject of SmithsonianImages.SI.Edu, a government ecommerce site built on a repository of 6,288 images of national significance. The site is breathtaking in scope, with imagery ranging from the historic cyanotypes of Edward Muybridge to historic photos from aviation, natural history, and many other fields. If the Smithsonian Institution is our attic, these photos are our collective scrapbook.

However, the web site imposes draconian limits on the use of this imagery. The site includes a copyright notice that to the layman would certainly discourage any use of the imagery. While personal, non-commercial use is purportedly allowed, it requires a half-dozen clicks before the user is allowed to download a low-resolution, watermarked image. An image without the watermark and at sufficient resolution to be useful requires a hefty fee, manual approval by the Smithsonian staff, and the resulting invoice specifically prohibits any further use without permission.

The letter goes into great detail about why they disagree with how things are being done – take a look if you are curious. Also -they didn’t just create this letter – they also created a free to download book titled Public Domain Prospectus which they declare as a tool for those researching the public domain status of the 6,288 images included (in their low resolution watermarked versions).

I went hunting on the Smithsonian Images site to see for myself. I found a few things. While the prices for prints or digital files do seem expensive to my eyes – there is the following note included in the Product and Pricing Information:

Special Note on Pricing: Smithsonian Photographic Services, as an instrument of the Smithsonian Institution, is a non-profit entity. Fees associated with the delivery of images represent material fees only and go to support the broader mission to create, archive, and preserve images associated with the Institution and it’s holdings.

That page also includes some information about how the images may be used, but for the full story I headed over to the Copyright Policy. That is when I started to get confused. The copyright policy on that page talks about “Use of text, images and other content on this website…”. Does that mean these same rules apply to the images you purchase as well?

Let’s take a closer look at one of the pages about a specific image. Here is a nice one of Fireworks over National Monuments. I click on the tempting ‘Download Image’ button and now I see more about what the Public.Resource.Org folks are talking about. One more click and I finally find what appears to be the official Commercial Use of Smithsonian Images page which concludes with:

Commercial distribution, publication or exploitation of Smithsonian files is specifically prohibited. Anyone wishing to use any of these files or images for commercial use or publication must first request and receive prior permission by contacting [Smithsonian Institution Office of Imaging & Photographic Services]. Permission for such use is granted on a case-by-case basis. A usage fee may be involved depending on the type and nature of the proposed use.

There is a special policy for school, teacher and student use of the watermarked versions of the images for free (with the right citations of course).

If I understand the Public.Resource.Org’s issues, it isn’t predominately with the price of the high resolution digital versions or even the print versions of these photos (though they DO touch on it in their letter and I think I side with Smithsonian Images on that aspect – it does cost money and time to make all that available). Rather it is with the firmness that Smithsonian Images claims that you must request permission to use any of the images you purchase for anything beyond personal or educational use. I think I like what NARA has on their website concerning the publication of their still photos which begins with these two paragraphs:

Generally, photographic records copied and sold by the National Archives and Records Administration (NARA) may be published without special permission or additional fees. NARA does not grant exclusive or non-exclusive publication privileges. Copies of Federal records, as part of the public domain, are equally available to all.

A portion of the photographs among our holdings are or may be subject to copyright restrictions. The National Archives does not confirm the copyright status of photographs, but will provide any information filed with the photograph. It is important to note that all of the digital images that are available on our website are in the public domain.

I can see how it might seem safer (from a “don’t sue us” point of view) to force a search by hand for each and every image as users request to use them. At the same time I would like to think that the folks over at Smithsonian Images already know which images are in the public domain. Maybe I am oversimplifying this, but I want to believe that the details of copyright are part of the metadata that could be supplied along with the date, photographer’s name and description.

I prefer the National Archives’ approach of stating clearly that they do not confirm the copyright status of photographs. They put it in the hands of the entity who wants to use the materials – though that might be small comfort to the average citizen not well versed in copyright rules.

The Wikipedia page on Copyright status of work by the U.S. government includes sections about digital historical material as well as work produced by government contractors. Reading through this makes me realize how quickly the copyright status of images such as those provided by Smithsonian Images and NARA can get confusing.

I think what Public.Resource.Org is doing with their propagation of public domain materials to locations where the public can actually get at them easily is interesting. I want to check back in a year and see how much they have set loose – and what materials they are asking for help to liberate. As I mentioned above, I think there could be some interesting models of individuals donating money to finance the digitization and of public domain materials. Something like what Fundable does to take pledges toward a specific fund-raising goal – and then only turn those pledges into funds if the goal is reached.

As for their great frustration with Smithsonian Images? Well, I see Public.Resource.Org’s side. In this age of Flickr.com – people are growing used to watching for Creative Commons Licenses. With so much out there with liberal Creative Commons Licenses and in the Public Domain, why struggle with images that are copyright protected unless you really need to?

I would like to think that rights management is one of the first things that would get sorted out before a large image collection is put online – especially if the goal is to produce a revenue stream. That said – I would love to know the real story here. I can imagine that the rights on many of those images are not clear cut. But if the Smithsonian Image people know that some of them are in the public domain – then why would they go through all that extra trouble to force a rights search for every image? Why not distinguish the ones which require research from those that don’t? Couldn’t it only help support the work of the Smithsonian to have their images used by as many projects as possible? Anyone reading this have an answer for us from the inside?

About the image above: Given that I prefer images without watermarks (as provided by Smithsonian Images) and that I know that the images on NARA’s site are in the public domain I went hunting for something pretty – and found the image I feature above. To find it yourself do a search for [Sunrise on Malibu Lake] in the Archival Research Catalog (ARC). These are the details included with the image:

Sunrise on Malibu Lake in the Santa Monica mountains near Malibu, California, which is located on the northwestern edge of Los Angeles County. The mountains contain the last semi-wilderness in Los Angeles County. This area so far has escaped development pressure. Some 84 percent of the state’s residents live within 30 miles of the coast and this concentration has resulted in increasing land use pressure. Several commissions have been authorized by the legislature to restrict coastal development, 05/1975.

Item from Record Group 412: Records of the Environmental Protection Agency, 1944 – 2000. NARA NAIL Control Number: NWDNS-412-DA-15109. Photograph by Charles O’Rear.

Metadata World Building: Freebase.com and OpenLibrary.org

July 17, 2007 10 Comments

I find it interesting to have discovered both Freebase.com (an open shared database of the world’s knowledge) and the Open Library Demo Site (ultimately intended to be a library of all the world’s information about all the world’s books) in the same week. They are both counting on crowdsourcing to populate large databases of information – all of which will be available for use by the general public. Freebase.com’s data is all licensed under Creative Commons CC-BY while Open Library describes that the data must be “a product of the people: letting them create and curate its catalog, contribute to its content, participate in its governance, and have full, free access to its data”.

For the Open Library project, the creators “wrote a new type of wiki that lets users enter structured data”. They have a page showing the data structure for bibliographic items on a Schema page. They also document their creation of a database framework called ThingDB which was designed to hold huge quantities of records, “hold arbitrary semi-structured data” and handle history/versioning.

For Freebase, the the data structure itself is evolving – and alpha users have a major hand in guiding that evolution. They have three major building blocks: Topics, Properties and Types. A Topic may be anything about which you want to create a Freebase entry. A person, place, thing, or idea. Properties are exactly what they sound like – such as the name of a person, the location of a place or the material a thing is made of. The Types are what make it all interesting. A Freebase Type is a set of properties. For any Topic that is about a person it makes sense that you want the easy option to add all the ‘person’ related Properties to that Topic easily. But a Topic can have more than one Type associated with it – and a Property can be populated by values that themselves are (or become) Topics. I would give you links to examples in Freebase, but to get your fingers in the mix at this point you need to put your name in the hat and wait for them to invite you to play with the alpha version.

Let us consider Books. Freebase already has the notion of an Author. Any Topic associated with the Author Type automagically can have books the person wrote associated with them as values for a multi-value property — and each of the books added will instantly become a new Topic (or you can pick them from a list if Freebase already knows about the book) with all the Properties you need for a book (and yes – it already knows who the Author is if you added the book from the Author’s page).

At the time I wrote this post, the Freebase topic page for Mark Twain shows the following associated types:

Person (People)
Film Writer (Film)
Author (Publishing)
Deceased Person (People)
Influence Node (mikelove’s types)
Book Subject (Publishing)

The values in parentheses above are the Domains to which a Type belongs. You will note that the ‘Influence Node’ Type is associated with a domain called mikelove’s types. This is because Freebase lets individuals create new Types. These types can later be promoted for use by anyone – and if I am reading the help properly, can be ‘published’ for others to use even before they are ‘promoted’ to be belong to an official Domain.

For Mark Twain, each of the Types listed above brings the opportunity to populated various Properties of structured data. Here are all the structured data Properties available for population for Mark Twain (some with their current values):

Name: Mark Twain
Description: currently imported from Wikipedia
Also known as: Samuel Langhorne Clemens
Gender: Male
Date of Birth: Nov 30, 1835
Place of Birth: Florida, Missouri
Country Of Nationality: United States
Profession
Religion
Spouse(s)
Parents: Jane Lampton Clemens, John Marshall Clemens,
Children
Sibling(s)
Height (meters)
Weight (kg)
Date of Death: 1910
Place of Death: Redding, Connecticut
Cause Of Death
Date of burial
Place of burial
Web Links
Employment History
Education
Quotations
Film Writing Credits
Books Written
Short Stories Written
Essays/Articles Written
Influenced By
Peers
Influenced
Books About This Topic
Short Works of Non-Fiction About This Topic

In contrast – take a look at the Open Library page for Mark Twain – these are the structured data elements I spotted (populated or otherwise) on that page:

Name
Text Entry
November 30, 1835 – April 21, 1910
Genres
Related Authors
Website
Location: Elmira, New York (buried)
Alternate Name: Samuel Clemens
Books by this author

I wonder how much of the book/publishing/literary world of data will be duplicated between Open Library and Freebase. They are both very young (‘alpha’ for Freebase vs ‘early technology preview’ for Open Library) and both are throwing open their arms for help. Open Library has a special page about the librarianship and another about how you can help. If you can get the secret pass into Freebase Alpha, there are plenty of discussions, examples, demos, help and opportunities to contribute. (I set the value of the Gender Property on the Mark Twain record while pondering it for this blog post).

The biggest difference between these two projects relates to what each is trying to accomplish. Freebase wants to be a super flexible universal database of knowledge that can be used to power applications (and they have tons of tools and APIs aimed at developers). Open Library is all about books, books, books – and all things related to them. Freebase has many Topics assigned the Book Type (8,749), but has even more associated with Person (355,359), Restaurant (99,971), City/Town (59,848), Company (20,405), Film (22,641) and many more.

Freebase is the growing creation of Metaweb (made up of veterans of Netscape, The Internet Archive, Alexa, Tellme, Intel and Broderbund) – . The Open Library Demo includes a tidy list of the people who created the it. I noticed Alexa and The Internet Archive on both lists – small world.

I love the idea of Open Library (and will enthusiastically follow its progress), but personally I am more excited to work on Archives related data in Freebase. I am pondering the creation of a new Archives Type and an Archival Collection or Archival Finding Aid Type. Sound like fun? Ask for an Alpha account and let me know when you get one … I would love to brainstorm this with others of a like mind.

International Environmental Data Rescue Organization: Rescuing At Risk Weather Records Around the World

June 7, 2007 2 Comments

In the middle of my crazy spring semester a few months back, I got a message about volunteer opportunities at the International Environmental Data Rescue Organization (IEDRO). I get emails from from VolunteerMatch.org every so often because I am always curious about virtual volunteer projects (ie, ways you can volunteer via your computer while in your pajamas). I filed the message away for when I actually had more time to take a closer look and it has finally made it to the top of my list.

A non-profit organization, IEDRO states their vision as being “.. to find, rescue, and digitize all historical environmental data and to make those data available to the world community.” They go on to explain on their website:

Old weather records are indeed worth the paper they are written on…actually tens of thousands times that value. These historic data are of critical importance to the countries within which they were taken, and to the world community as well. Yet, millions of these old records have already perished with the valuable information contained within, lost forever. These unique records, some dating back to the 1500s, now reside on paper at great risk from mold, mildew, fire, vermin, and old age (paper and ink deteriorate) or being tossed away because of lack of storage space. Once these data are lost, they are lost forever. There are no back up sources; nothing in reserve.

Why are these weather records valuable? IEDRO gives lots of great examples. Old weather records can:

inform the construction and engineering community about maximum winds recorded, temperature extremes, rainfall and floods
let farmers know the true frequency of drought, flood, extreme temperatures and in some areas, the amount of sunshine enabling them to better plan crop varieties and irrigation or drainage systems increasing their food production and helping to alleviate hunger.
assist in explaining historical events such as plague and famine, movement of cultures, insect movements (i.e. locusts in Africa), and are used in epidemiological studies.
provide our global climate computer models with baseline information enabling them to better predict seasonal extremes. This provides more accurate real-time forecasts and warnings and a better understanding of global change and validation of global warming.

The IEDRO site includes excellent scenarios in which accurate historical weather data can help save lives. You can read about the subsistence farmer who doesn’t understand the frequency of droughts well enough to make good choices about the kind of rice he plants, the way that weather impacts the vectorization models of diseases such as malaria and about the computer programs that need historical weather data to accurately predict floods. I also found this Global Hazards and Extremes page on the NCDC’s site – and I wonder what sorts of maps they could make about the weather one or two hundred years ago if all the historical climate data records were already available.

There was additional information available on IEDRO’s VolunteerMatch page. Another activity they list for their organization is: “Negotiating with foreign national meteorological services for IEDRO access to their original observations or microfilm/microfiche or magnetic copies of those observations and gaining their unrestricted permission to make copies of those data”.

IEDRO is making it their business to coordinate efforts in multiple countries to find and take digital photos of at risk weather records. They include information on their website about their data rescue process. I love their advice about being tenacious and creative when considering where these weather records might be found. Don’t only look at the national meteorological services! Consider airports, military sites, museums, private homes and church archives. The most unusual location logged so far was a monastery in Chile.

Once the records are located, each record is photographed with a digital camera. They have a special page showing examples of bad digital photos to help those taking the digital photos in the field, as well as a guidelines and procedures document available in PDF (and therefore easy to print and use as reference offline).

The digital images of the rescued records are then sent to NOAA’s National Climatic Data Center (NCDC) in Asheville, North Carolina. The NCDC is part of the National Environmental Satellite, Data and Information Service (NESDIS) which is in turn under the umbrella of the National Oceanic and Atmospheric Administration (NOAA). The NCDC’s website claims they have the “World’s Largest Archive of Climate Data”. The NCDC has people contracted to transcribe the data and ensure the preservation of the digital image copies. Finally, the data will be made available to the world.

IEDRO already lists these ten countries as locations where activities are underway: Kenya, Malawi, Mozambique, Niger, Senegal, Zambia, Chile, Uruguay, Dominican Republic and Nicaragua.

I am fascinated by this organization. On a personal level it brings together a lot of things I am interested in – archives, the environment, GIS data, temporal data and an interesting use of technology. This is such a great example of records that might seem unimportant – but turn out to be crucial to improving lives in the here and now. It shows the need for international cooperation, good technical training and being proactive. I know that a lot of archivists would consider this more of a scientific research mission (the goal here is to get that data for the purposes of research), but no matter what else these are – they are still archival records.

reCAPTCHA: crowdsourcing transcription comes to life

May 28, 2007 5 Comments

With a tag-line like ‘Stop Spam, Read Books’ – how can you not love reCAPTCHA? You might have already read about it on Boing Boing , NetworkWorld.com or digitizationblog – but I just couldn’t let it go by without talking about it.

Haven’t heard about reCAPTCHA yet? Ok.. have you ever filled out an online form that made you look at an image and type the letters or numbers that you see? These ‘verify you are a human’ sorts of challenges are used everywhere from on-line concert ticket purchase sites who don’t want scalpers to get too many of the tickets to blogs that are trying to prevent spam. What reCAPTCHA has done is harness this user effort to assist in the transcription of hard to OCR text from digitized books in the Internet Archive. Their website has a great explanation about what they are doing – and they include this great graphic below to show why human intervention is needed.

reCAPTCHA shows two words for each challenge – one that it knows the transcription of and a second that needs human verification. Slowly but surely all the words OCR doesn’t understand get transcribed and made available for indexing and search.

I have posted before about ideas for transcription using the power of many hands and eyes (see Archival Transcriptions: for the public, by the public) – but my ideas were more along the lines of what the genealogists are doing on sites like USGenWeb. It is so exciting to me that a version of this is out there – and I LOVE their take on it. Rather than find people who want to do transcription, they have taken an action lots of folks are already used to performing and given it more purpose. The statistics behind this are powerful. Apparently 60 million of these challenges are entered every DAY.

Want to try it? Leave a comment on this post (or any post in my blog) and you will get to see and use reCAPTCHA. I can also testify that the installation of this on a WordPress blog is well documented, fast and easy.

ArchivesZ: Visualizing Archival Collections

May 13, 2007 8 Comments

Announcing ArchivesZ – a tool for visualizing archival collections. This prototype is the final project for my information visualization class. It is a web based tool designed to support exploration of aggregated data about archival collections – inspired by the availability of structured data in EAD encoded finding aids.

For visual thinkers who just want to see what this is about – take a look at our 5 minute video demonstration. I don’t yet have a version online for folks to play with – but that is in the works.

This is the official blurb we came up with to describe the project:

ArchivesZ is an information visualization tool designed to support search, understanding and exploration of of archival and manuscript collections. The tool addresses one of the major challenges facing those who work with archival records – the need to understand the scope and quantity of available records. Since archival collections are unique, vary dramatically in record quantity and are organized based on the records creators it can be a great challenge for users to gain perspective concerning the available records across multiple collections. ArchivesZ leverages a unique dual sided histogram to support exploration of the multiple subjects assigned to each collection. As subject terms are selected, the dual sided histogram chart is generated to display related subjects. The tool combines the dual sided histogram with a more traditional histogram displaying year data to permit tightly coupled, multi-dimensional browsing of subject and time period metadata. By representing the distribution of subjects and time periods using the metric of total aggregate linear feet of associated collections, ArchivesZ permits users to get a better sense of total available research materials than they would by viewing a standard search result list.

If you are curious about what a ‘dual-sided histogram’ actually is (or just want to read more about the process and ideas that led us to the current incarnation of ArchivesZ) take a look at our final paper about ArchivesZ.

There is a very long list of features I would like to add or improve but of course there is only so much you can do in the few weeks available for a project like this. Some of our ideas are detailed at the end of the paper I linked to above. I plan to continue working on ArchivesZ and I welcome all feedback – either as comments to this post or via email to jeanne AT spellboundblog.com.

Copyright Law: Archives, Digital Materials and Section 108

April 12, 2007 2 Comments

I just found my way today to Copysense (obviously I don’t have enough feeds to read as it is!). Their current clippings post highlighted part of the following quote as their Quote of the Week.

“[L]egislative changes to the copyright law are needed. First, we need to amend the law to give the Library of Congress additional flexibility to acquire the digital version of a work that best meets the Library’s future needs, even if that edition has not been made available to the public. Second, section 108 of the law, which provides limited exceptions for libraries and archives, does not adequately address many of the issues unique to digital media—not from the perspective of copyright owners; not from the perspective of libraries and archives.” Marybeth Peters , Register of Copyrights, March 20, 2007

Marybeth Peters was speaking to the Subcommittee on Legislative Branch of the Committee on Appropriations about the Future of Digital Libraries.

Copysense makes some great points about the quote:

Two things strike us as interesting about Ms. Peters’ quote. First, she makes the quote while The Section 108 Study Group continues to work through some very thorny issues related to the statutes application in the digital age […] Second, while Peters’ quote articulates what most information professionals involved in copyright think is obvious, her comments suggest that only recently is she acknowledging the effect of copyright law on this nation’s de facto national library. […] [S]omehow it seems that Ms. Peters is just now beginning to realize that as the Library of Congress gets involved in the digitization and digital work so many other libraries already are involved in, that august institution also may be hamstrung by copyright.

I did my best to read through Section 108 of the Copyright Law – subtitled “Limitations on exclusive rights: Reproduction by libraries and archives”. I found it hard to get my head around … definitely stiff going. There are 9 different subsections (‘a’ through ‘i’) each with there own numbered exceptions or requirements. Anxious to get a grasp on what this all really means – I found LLRX.com and their Library Digitization Projects and Copyright page. This definitely was an easier read and helped me get further in my understanding of the current rules.

Next I explored the website for the Section 108 Study Group that is hard at work figuring out what a good new version of Section 108 would look like. I particularly like the overview on the About page. They have a 32 page document titled Overview of the Libraries and Archives Exception in the Copyright Act: Background, History, and Meaning for those of you who want the full 9 years on what has gotten us to where we are today with Section 108.

For a taste of current opinions – go to the Public Comments page which provides links to all the written responses submitted to the Notice of public roundtable with request for comments. There are clear representatives from many sides of the issue. I spotted responses from SAA, ALA and ARL as well as from MPAA, AAP and RIAA. All told there are 35 responses (and no, I didn’t read them all). I was more interested in all the different groups and individuals that took the time to write and send comments (and a lot of time at that – considering the complicated nature of the original request for comments and the length of the comments themselves). I was also intrigued to see the wide array of job titles of the authors. These are leaders and policy makers (and their lawyers) making sure their organizations’ opinions are included in this discussion.

Next stop – the Public Roundtables page with it’s links to transcripts from the roundtables – including the most recent one held January 31, 2007. Thanks to the magic of Victoria’s Transcription Services, the full transcripts of the roundtables are online. No, I haven’t read all of these either. I did skim through a bit of it to get a taste of the discussions – and there is some great stuff here. Lots of people who really care about the issues carefully and respectfully exploring the nitty-gritty details to try and reach good compromises. This is definitely on my ‘bookmark to read later’ list.

Karen Coyle has a nice post over on Coyle’s InFormation that includes all sorts of excerpts from the transcripts. It gives you a good flavor of what some of these conversations are like – so many people in the same room with such different frames of reference.

This is not easy stuff. There is no simple answer. It will be interesting to see what shape the next version of Section 108 takes with so many people with very different priorities pulling in so many directions.

The good news is that there are people with the patience and dedication to carefully gather feedback, hold roundtables and create recommendations. Hurrah for the hard working members of the Section 108 Study Group – all 19 of them!

Visualizing Archival Collections

April 8, 2007 1 Comment

As I mentioned earlier, I am taking an Information Visualization class this term. For our final class project I managed to inspire two other classmates to join me in creating a visualization tool based on the structured data found in the XML version of EAD finding aids.

We started with the XML of the EAD finding aids from University of Maryland’s ArchivesUM and the Library of Congress Finding Aids. My teammates have written a parser that extracts various things from the XML such as title, collection size, inclusive dates and subjects. Our goal is to create an innovative way to improve the exploration and understanding of archival collections using an interactive visualization.

Our main targets right now are to use a combination of subjects, years and collection size to give users a better impression of the quantity of archival materials that fit various search criteria. I am a bit obsessed about using the collection size as a metric for helping users understand the quantity of materials. If you do a search for a book in a library’s catalog – getting 20 hits usually means that you are considering 20 books. If you consider archival collections – 20 hits could mean 20 linear feet (20 collections each of which is 1 linear foot in size) or it could mean 2000 linear feet (20 collections each of which is 100 linear feet in size). Understanding this difference is something that visualization can help us with. Rather than communicating only the number of results – the visualization will communicate the total size of collections assigned each of the various subjects.

I have uploaded 2 preliminary screen mockups one here and the second here trying to get at my ideas for how this might work.

Not reflected in the mock-ups is what could happen when a user clicks on the ‘related subject’ bars. Depending on where they click – one of two things could happen. If they click on the ‘related subject’ bar WITHIN the boundaries of the selected subject (in the case above, that would mean within the ‘Maryland’ box), then the search would filter further to only show those collections that have both the ‘Maryland’ and newly ‘added’ tag. The ‘related subjects’ list and displayed year distribution would change accordingly as well. If, instead, the user clicks on a ‘related subject’ bar OUTSIDE the boundary of the selected subject — then that subject would become the new (and only) selected subject and the displayed collections, related subjects and years would change accordingly.

So that is what we have so far. If you want to keep an eye on our progress, our team has a page up on our class wiki about this project. I have a ton of ideas of other things I would love to add to this (my favorite being a map of the world with indications of where the largest amount of archival materials can be found based on a keyword or subject search) – but we have to keep our feet on the ground long enough actually build something for our class project. This is probably a good thing. Smaller goals make for a greater chance of success.

Getting Your Toes Wet: Basic Principals of Design for the New Web

April 5, 2007

Ellyssa Kroski of InfoTangle has created a great overview of current trends in website and application design in her post Information Design for the New Web. If you are going to Computers in Libraries, you can see her present the ideas she discusses in her post in a session of the same name on Monday April 16.

She highlights 3 core principles with clear explanations and great examples:

Keep it Simple
Make it Social
Offer Alternate Navigation

As archives continue to dive into the deep end of the internet pool, more and more archivists will find themselves participating in discussions about website design choices. Understanding basic principals like those discussed in Kroski’s post will go a long way to making archivists feel more comfortable contributing to these sorts of discussions.

Don’t think that things like this should be left to the IT department or only the ‘techie archivists’ on your staff (if you have any). You all have a lot to contribute. You know your collections. You know the importance of archival principals of provenance, original order and context. There are lots of aspects of archival materials that traditional web designers might not consider important. Things that you know are very important if people are to understand your archives’ materials while browsing from the comfort of their homes.

So dip your toes in. Learn some buzz words, look at some fun websites and get comfortable with some innovative ideas. The water is just fine.

Considering Historians, Archivists and Born Digital Records

March 23, 2007 3 Comments

I think I renamed this post at least 12 times. My original intention was was to consider the impact of born digital records on the skills needed for the historian/researchers of the future. In addition I found myself exploring the dividing lines among a number of possible roles in ensuring access to the information written in the 1s and 0s of our born digital records.

After my last post about the impact of anonymization of Google Logs, a friend directed me to the work of Dr. Latanya Sweeney. Reading through the information about her research I found Trail Re-identification: Learning Who You are From Where You Have Been. Given enough data to work with, algorithms can be written that often can re-identify the individuals who performed the original searches. Carnegie Mellon University‘s Data Privacy Lab includes the Trails Learning Project with the goal of answering the question “How can people be identified to the trail of seemingly innocent and anonymous data they leave behind at different locations?”. So it seems that there may be a lot of born digital records that start out anonymous but that may permit ‘re-identification’ – given the application of the right tools or techniques. That is fine – historians have often needed to become detectives. They have spent years developing techniques for the analysis of paper documents to support ‘re-identification’. Who wrote this letter? Is this document real or a forgery? Who is the ‘Mildred’ referenced in this record?

The field of diplomatics studies the authenticity and provenance of documents by looking at everything from the paper they were written on to the style of writing to the ink used. I like the idea of using the term ‘digital diplomatics’ for the ever increasing process of verifying and validating born digital records. Google found me the Digital Diplomatics conference that took place earlier this year in Munich. Unfortunately it was more geared toward investigating how the use of computers can enhance traditional diplomatic approaches rather than how to authenticate the provenance of born digital records.

In the March 2007 issue of Scientific American I found the article A Digital Life. It talks primarily about the Microsoft Research project MyLifeBits. A team at Microsoft Research has spent the last six years creating what they call a ‘digital personal archive’ of team member Gordon Bell. This archive hopes to “record all of Bell’s communications with other people and machines, as well as the images he sees, the sounds he hears and the Web sites he visits–storing everything in a personal digital archive that is both searchable and secure.”

They are not blind to the long term challenges of preserving the data itself in some accessible format:

Digital archivists will have to constantly convert their files to the latest formats, and in some cases they may need to run emulators of older machines to retrieve the data. A small industry will probably emerge just to keep people from losing information because of format evolution.

The article concludes:

Digital memories will yield benefits in a wide spectrum of areas, providing treasure troves of information about how people think and feel. By constantly monitoring the health of their patients, future doctors may develop better treatments for heart disease, cancer and other illnesses. Scientists will be able to get a glimpse into the thought processes of their predecessors, and future historians will be able to examine the past in unprecedented detail. The opportunities are restricted only by our ability to imagine them.

Historians will have at least these two types of digital artifacts to explore – those gathered purposefully (such as the digital personal archives described above) and those generated as a byproduct of other activity (such as the Google search logs). Might these be the future parallels to the ‘manuscript’ and ‘corporate’ archives of today?

So we have both the ideas of the Digital Archivist and the Digital Historian. What about a Digital Archaeologist? I am not the first to ponder the possible future job of Digital Archaeologist. A bit of googling of the term led me to Dark Star Gazette and Dear Digital Archaeologist. Back in February of 2007 they pondered:

Will there be digital archaeologists, people who sift through our society’s discarded files and broken web links, carefully brushing away revisions and piecing together antiquated file formats? Will a team of grad students working on their PhDs a thousand, or two thousand, years from now be digging through old blog entries, still archived online in some remote descendant of the Wayback Machine or a copy of Google’s backup tapes?

I can only imagine a world in which this is in fact the case. Given that premise, at what point does the historian get too far from the primary source? If the historian does not understand exactly what a computer program does to extract the information they want from logs or ‘digital memory repositories’ – are they no longer working with the primary source?

Imagine any field in which historians do research. Music? Accounting? Science? In order examine and interpret primary source records a historian becomes something of an expert in that field. Consider the historian documenting the life of a famous scientist based partly on their lab notebooks. That historian would be best served by being taught how to interpret the notebooks themselves. The historian must be fluent in the language of the record in order to gain the most direct access to the information.

Ah – but if there really are Digital Archaeologists in the far future, perhaps they would be the connection between the primary source born digital records and the historians who wish to study them. Or perhaps the Digital Archivist, in a new take on ‘arranging records’, would transform digital chaos into meaningful records for use by researchers? The field of expertise on the historians part would need only be in the content of the records – not exactly how they were rescued from the digital abyss.

Would a Digital Historian be someone who only considers the history of the digital landscape or a historian especially well versed in the interpretation of digital records? In Daniel Cohen and Roy Rosenzweig‘s book Digital History: A Guide to Gathering, Preserving, And Presenting the Past on the Web they seem to use the term in the present tense to refer to historians who uses computers and technology to support and expand the reach of their research. Yet, in his essay Scarcity or Abundance? Preserving the Past in a Digital Era, Roy Rosenzweig proposes:

Future graduate programs will probably have to teach such social-scientific and quantitative methods as well as such other skills as “digital archaeology”(the ability to “read” arcane computer formats), “digital diplomatics” (the modern version of the old science of authenticating documents), and data mining (the ability to find the historical needle in the digital hay). In the coming years, “contemporary historians” may need more specialized research and “language” skills than medievalists do.

What is my imagined skill set for the historian of our digital world? A willingness to dig into the rich and chaotic world of born digital records. The ability to use tools and find partners to assist in the interpretation of those records. Equal comfort working at tables covered in dusty boxes and in the virtual domain of glowing computer terminals. And of course – the same curiosity and sense of adventure that has always drawn people to the path of being a historian.

We cannot predict the future – we can only do our best to adapt to what we see before us. I suspect the prefixing of every job title with the word ‘digital’ will disappear over time – much as the prefixing of everything with the letter ‘e’ to let you know that something was electronic or online has ebbed out of popular culture. As the historians and archivists of today evolve into the historians and archivists of tomorrow they will have to deal with born digital records – no matter what job title we give them.

Category: access