Since the early 1990s, the digital library field has emerged as an important area of research and development. There are hundreds of projects and thousands of reports/publications describing them. One of the active research programs is in the USA, funded by the National Science Foundation.  A summary of that Digital Libraries Initiative makes clear the broad scope of work underway (Fox 1999a). NSF has deliberately selected a diverse set of content areas, genre, media, and user communities in an effort to rapidly develop the field (Lesk 1999). In the following sections we explore the overall process of such development.

Technology

     Work on digital libraries has been facilitated by technical advances in many areas. For the first time, storage systems are readily affordable that can handle enormous text collections, very large image collections, and large audio or video collections. Fast processors, supercomputers, cluster-computers, networks of workstations, and other computational aids have provided ample processing capacity to handle user communities operating on a global scale. Increases in network speeds and bandwidth have made it possible to build distributed systems that perform well and have high reliability. Computationally expensive algorithms have been refined so that useful techniques such as LSI can help with multilingual retrieval and other applications (Dumais 1998). High-end graphics systems and virtual environments also have evolved to be usable for information visualization as well as interfacing with digital libraries (Das Neves & Fox 2000). Representation schemes like PDF and XML have made digital documents easy to produce and share; facilitating information interchange and encouraging further digitization.

      An example of the effects of technology can be seen with regard to the MARIAN digital library system developed at Virginia Tech (Can, Fox, Snavely & France 1995; Fox, France, Sahle, Daoud & Cline 1993a; France, 2000; Zhao, 1999). Over the last decade, coding has switched from C to C++ to Java. Hardware has switched from mainframes to minis to PCs. Current work to enhance performance includes development of new algorithms to manipulate inverted files (Sornil 2000) on the Virginia Tech PetaPlex system, which has 100 processors and 2.5 terabytes of disk storage capacity. Earlier studies of performance (Zhao 1999) demonstrated that the architecture is scalable, and showed that, for example, changing some of the internal communication from TCP to UDP would lead to substantive improvement.

Economic, Social, and Legal Issues

     According to the 5S framework, the topmost level deals with “societies”. Though technology has made possible many advances in digital libraries, all such efforts are situated in a social context, as can be seen in Figure 1.  Many of the key social issues were identified in a NSF-funded workshop on this topic (Borgman 1996). A good explanation of social issues in constructing and evaluating digital libraries appears in (Kilker & Gay 1998).

     
      Underlying work on the Networked Digital Library of Theses and Dissertations (discussed further in Chapter 5) is a strong social and educational rationale, to prepare the next generation of scholars for the Information Age (Fox 1997; Fox, 1998a; Fox et al. 1996; Fox, Hall, & Kipp 1997a; Fox et al. 1997b). Its aims – of encouraging discussion about intellectual property rights among students and faculty, of building awareness and infrastructure about digital libraries on campuses, and of developing a new genre for communication among graduate students and researchers – are largely being met. The impact of economics is remarkable, in that making works available for free leads to hundreds or thousands of downloads per work per year. This contrasts with interlibrary loan or buying copies for roughly $50, which very rarely led to more than 5 accesses per year.

     
      Social and technical issues often relate. For example, the culture and social atmosphere determines how electronic theses and dissertations will be managed on a particular campus, or across broader boundaries involving states or nations. Universities with advanced infrastructure, like Caltech, MIT, and Virginia Tech, have their own services. On the other hand, there are regional/national projects associated with NDLTD in Ohio, Catalunya, Australia, Germany, India, Portugal, and South Africa. Because of this arrangement, a federated search mechanism was implemented (Powell & Fox 1998), though future plans call for use of Open Archives and regular harvesting. Groups that own content or have a tradition of managing it can continue to do so, while at the same time technical approaches can allow comprehensive search across such distributed collections.

      Legal issues also become more visible with digital libraries. There often was little concern over copyright issues when preparing a dissertation that would largely go unread, sitting on a shelf in a local library. However, with the potential of thousands of downloads from around the world, authors must be very careful not to include a copyrighted image or other content without permission.

      Other economic, social, and legal issues have come to the fore with digital libraries. In many states, like Virginia, a single university library can run a service for the whole state, further extending the benefits of volume purchasing. On the other hand, many such services involve a contract with an information provider that often has a complex set of terms and conditions, meaning that libraries now require more legal counsel.

      Digital libraries allow new groups to assemble around new collections of interest. In many countries, digital libraries show promise regarding preserving cultural, historic, and linguistic records. In a number of situations, they have the potential of aiding economic development by supporting localized and distance education. We explore such cases among those discussed in the next subsection.

Initiatives and Projects

      There are hundreds of digital library efforts underway around the world. A good place to find out about many of these is in D-Lib Magazine (Arms 2000a), which by the end of 2000 has about 1,000 entries.

      As mentioned earlier, the National Science Foundation in USA has funded the Digital Libraries Initiative, with awards granted in 1994, 1998, and 1999.  Well over $50M has gone to a wide range of research projects.  NSF support to Stanford University through this program has led in part to the appearance of Google, one of the most popular search services on WWW. Other technical advances have led to methods for processing and searching collections of images, music, and video. A great expansion in work on geographic information systems and other spatial data has also resulted. Microsoft’s TerraServer (http://terraserver.microsoft.com/) demonstrates how very large data collections can serve huge user communities in spite of using relatively modest hardware resources.

      The museum community has demonstrated methods of accessing distributed cultural heritage information (Moen 1998). Sacred and/or precious resources, including antiquities, have become available from sources such as the Vatican (Gladney, Mintzer, Schiattarella, Bescos, & Treu 1998; Mintzer et al. 1996). Special collections, such as of butterflies (Hong, Chen, & Hsiang, 2000) or floristic information (Amavizca, Sánchez, & Abascal 1999; Sánchez, Fernández, & Schnase 1998a; Sánchez et al. 1999; Sánchez et al. 1998b), have become accessible through digital libraries.

      In some ways the broadest and most influential digital library efforts have to do with education. For example, in USA, after years of study, it was decided that NSF should support development of a digital library to support learning by students (Arms 1999). This has led to over $50M committed to the National Science (and Mathematics, Engineering, and Technology Education) Digital Library, which should be launched by the end of 2002 (www.smete.org 2000). Many thousands of teachers, and millions of students, will benefit from this resource as it aims to enhance learning through dynamic interaction, visualization, simulation, and other computer-related devices.