Evaluating DH Work: Guidelines for Librarians 2

In this post, Zach Coble explores the benefits of creating guidelines for the evaluation of librarians’ digital humanities work for the purposes of hiring, appointment, tenure, and promotion, and offers a basic framework for what those guidelines might look like.

This post was published in the Journal of Digital Humanities, volume 1, issue 4.

Digital humanities (DH), as well as related fields such digital media studies and digital libraries, have presented many opportunities for libraries. These include the establishment of DH centers, the development of new data standards, new forms of scholarly communication, the creation of new resources (and novel ways of asking questions of those resources), and the development of new tools for scholarship and accessing collections.[1] However, traditional modes of evaluation do not address many of the key aspects of DH work. As librarians become more involved in DH and begin to take on the title of “Digital Humanities Librarian,” how can we ensure that their work will be appropriately reviewed? While some librarians work individually on personal DH projects or scholarship, most collaborate with faculty, fellow librarians, and information technologists across campus and across institutions. The collaborative nature of DH work often blurs the lines when it comes to defining individual’s responsibilities and contributions. Similarly, new forms of scholarly output, such as a website rather than a paper or presentation, present additional challenges for those tasked with evaluating DH work.

Written guidelines for evaluation ensure that projects are reviewed fairly and provide a clear path for job hiring and advancement. Libraries clearly understand the importance of assessment and evaluation. The ACRL has guidelines for the evaluation of tenure track librarians and for those without faculty status. In 2010, Megan Oakleaf made waves with her Value of Academic Libraries report, which utilized existing assessment measures, such as college students’ information literacy skills, to demonstrate the positive impact of libraries. As the field of DH continues to grow, libraries will increasingly be called upon to dedicate time and resources to supporting this work. In order to encourage more libraries to support DH, to provide a framework that will encourage individual librarians to participate in DH, and to acknowledge and reward excellent work, libraries should develop guidelines for evaluation of librarians engaging in DH work.

Although librarians are often cited as important collaborators in DH projects, librarianship as a profession lacks a coordinated approach to DH. There are many reasons for this, such as the broad interdisciplinarity and rapidly evolving nature of DH, which makes it difficult to articulate a large-scale response. Yet it also stems from the fact that library involvement in DH varies across institutions: some libraries at large research-intensive universities host active DH centers while many small schools (as well as public libraries, special libraries, and so forth) are only vaguely aware of DH, if at all. [pullquote]A framework for evaluating DH work performed by librarians would ideally be one piece of a program to address DH from libraries.[/pullquote]In a recent survey by the Association of College and Research Libraries Digital Humanities Discussion Group, most of the librarians who responded did not have DH in their job title or description. Equally diverse are the types of work that librarians contribute to DH projects. A 2011 report on DH in libraries by the Association of Research Libraries (ARL) noted that DH projects often call upon librarians for consultation and project management, technical and metadata support, instructional services, and resource identification.

A framework for evaluating DH work performed by librarians would ideally be one piece of a program to address DH from libraries. Such a program, possibly from the Association of College and Research Libraries, might also include criteria for undertaking digital projects and best practices for doing DH work. As the 2011 ARL report notes, “The general lack of policies, protocols, and procedures has resulted in a slow and, at times, frustrating experience for both library staff and scholars. This points toward the need for libraries to coordinate their efforts as demand for such collaborative projects increases.” Without an organized response, librarians lack the incentives, resource support, institutional backing, and network of colleagues necessary to be successful.[2] On the other hand, a coordinated approach could encourage more librarians to get involved in DH, motivate individual libraries to adopt related policies specific to their local needs, foster greater participation among libraries in the DH community, and create the demand for increased training opportunities–both as continuing education for professionals and in library schools.

Other organizations, such as the Modern Language Association, NINES, and 18thConnect, have recognized the distinct nature of DH work and adopted separate guidelines for the evaluation of digital projects.[3] Libraries would benefit from having a similar set of guidelines. Of course, every institution is different and no one set of guidelines will work for everyone. Also, the context and scope of a librarian’s contribution should be taken into account–a librarian asked to consult on metadata standards should not be faulted if the project fails to follow web design best practices. While acknowledging such nuances, there are certain baseline ideas that should be addressed. The following list draws upon existing guidelines for the evaluation of DH work and incorporates additional elements specific to libraries. It is intended to help generate conversation and is not meant to be comprehensive.

Peer Review
  • Traditional concepts of peer review still apply: review projects for impact, intended audience, originality, and excellence based on content, form, interpretation.
  • There are peer review groups specifically for DH projects (e.g. NINES, 18thConnect, MLA Committee on Scholarly Editions); qualified librarians could seek to join these groups or possibly to create a similar group comprised of librarians.
Nature of Digital Projects
  • How does the digital component contribute something that couldn’t otherwise be communicated?
  • The project should be evaluated in the medium in which it was created.
  • Reviewers should acknowledge the ongoing nature of digital projects (i.e. lack of a “finished product”).
  • Did the project consult outside experts to assess the project’s content and technical structure?
  • How does the project relate with other digital scholarship projects?
  • Is there an intentional and appropriate organization of information?
  • Does the project use accepted standards for web design, metadata, and encoding?
  • Is there interoperability with other sites, such as OAI-PMH?
  • Is there a thoughtful balance between design, content, and medium?
  • How does the project address issues of digital preservation?
  • Is there documentation or is the site code made available?
Other Considerations
  • Was the project grant funded?
  • Did the project result in any conference presentations or print publications?

References and Further Reading

[2]     For a discussion of barriers to DH in libraries, see Miriam Posner, “What are some challenges to doing DH in the library?” ; Trevor Munoz, “Digital humanities in the library isn’t a service“; Mike Furlough, “Some Institutional Challenges to Supporting DH in the Library“; The Library Loon, “Additional hurdles to novel library services.”
[3]     See also University of Florida George A. Smathers Libraries, Facilitated Peer Review Committee; Roy Rosenzweig Center for History and New Media, Zotero | Groups > Digital Humanities > Library > Assessment and Evaluation; Modern Language Association, Guidelines for Editors of Scholarly Editions; University of Nebraska-Lincoln Center for Digital Research in the Humanities, Promotion & Tenure Criteria for Assessing Digital Research in the Humanities.

Zach Coble

Zach is the Digital Scholarship Specialist at New York University.

2 thoughts on “Evaluating DH Work: Guidelines for Librarians

  1. Pingback: New Resources for the Evaluation of Digital Scholarship | laurie n. taylor

  2. Pingback: grasping at a grasp of project management | the girl works

Comments are closed.