Research articlesThe benefits of ‘slow’ development: towards a best practice for sustainable technical infrastructure through the Davy Notebooks Project
Abstract
In this article we consider technical development and its role in digital humanities research efforts. We critique the concept that ‘novel’ development is crucial for innovation and tie this thinking to the corporatization of higher education. We suggest instead that sustainable technical development practices require a combination of need, creativity and reuse of existing technical infrastructure. First, we present our theory of ‘slow’ development based on this need-driven approach, then demonstrate how this theory can be applied to digital humanities research efforts using the Davy Notebooks Project as a case study. By tracing the history of this multi-phase, public crowdsourcing project and exposing the decision-making process behind its technical development, we demonstrate the promise and possibility of the ‘slow’ method proposed. When read in conjunction with the other essays in this special issue, we hope that this article will demonstrate how digital humanities methods and technical infrastructure can support and sustain traditional modes of scholarship, and the importance of applying the same careful approach to technical development that is applied to research methods more broadly.
Introduction
This article will chronicle the development approach implemented for digital humanities tools and infrastructure on the Zooniverse crowdsourcing platform. Zooniverse is the world's largest platform for crowdsourced research—at the time of writing, more than 450 projects have publicly launched since the platform was created in 2009. A robust community of more than 2.6 million registered volunteers (at the time of writing) grows larger every day. Using the Davy Notebooks Project as a case study,1 this article will describe the process used to move from a small pilot project run by Lancaster University to an international multi-institutional collaboration crowdsourcing the transcription of 83 handwritten notebooks of Sir Humphry Davy (1778–1829).
In particular, it will articulate the process of ‘slow’ development used to ensure that the project contributed sustainable technical output while supporting the needs and requirements of the research and software development teams and helping to ensure an ethical approach to working with volunteers.2 Ultimately, this article aims to demonstrate that intentional development, centred on data- and community-driven requirements, will produce sustainable tools that are more impactful in the long term than their novel counterparts, produced for the sake of innovation.
In the field of cultural heritage, crowdsourcing is ‘a form of digitally enabled participation that promises deeper, more engaged relationships with the public via meaningful tasks’.3 For such crowdsourcing activities to take place online, researchers must either create custom web infrastructure (from scratch or through adapting off-the-shelf tools) or use existing technical platforms to achieve their goals. In the field of cultural heritage, crowdsourcing goals usually involve both data transformation and public engagement.
As crowdsourcing has grown in popularity over the past two decades, the available technical options have increased, reducing the need for crowdsourcing projects to be created exclusively through custom development. There are now multiple resources that cultural heritage institutions can use to invite the public into the process of data transformation, collection and editing. These tools exist in a range of options for access, from paid models to free and open source, to a combination of the two: some platforms offer free trials, for example, and then charge a fee for additional use beyond the trial period or data limit.
For platform maintainers, particularly those who maintain free and open-source software (like Zooniverse), the challenge then becomes how to keep tools from becoming obsolete, while sustaining those that already work and are widely used. For researchers, the task is to identify which of the technical options best fit the goals and objectives of their project. To meet the goals of both communities, platform maintainers and researchers must communicate about their project aims, technical requirements and the limitations and challenges that come along with the process.
Defining ‘slow’ development: inspiration and application within digital humanities practice
For the purposes of technical development in the field of digital humanities, this article will both propose and use the following definition: ‘slow’ development is an approach to technical creation and project scalability that prioritizes people, reusability and sustainability, and ethical collaboration. It allows teams to identify what they need to create and what they can reuse, and to consider how doing less can result in work that is useful to a range of people far beyond its original use case. The scare quotes around ‘slow’ are intended to point out that the term as it is used here does not mean taking place over an extended period. Instead, ‘slow’ refers to a careful and care-based approach to development and innovation that is based in strategic planning for long-term maintenance and (re)use of tools and infrastructure.
The concept of ‘slow’ development as it is used in this article is directly inspired by the practice of slow librarianship, proposed in 2017 by Julia Glassman, who based it on Carlo Petrini's slow food movement that began in the 1980s and called it a response to the ‘innovation fetish’—in particular, the concept of innovation for the sake of innovation, which she connects to the growing trend of ‘deeply corporatized academia’.4 In library scholarship, the term was later picked up by Meredith Farkas, who has written in detail on the topic,5 following Glassman's initial inspiration and incorporating other more recent sources including Dean Spade's writing on mutual aid, Adrienne Maree Brown's on antiracism and Jenny Odell's work on the attention economy.6
The work here is aligned in an academic context with The Great Lakes Feminist Geography Collective's ‘For slow scholarship: a feminist politics of resistance through collective action in the neoliberal university’, which in 2015 joined a growing list of slow-advocating publications in their call to ‘challeng[e] the growing inequities in higher education’ through slowing down scholarship and rejecting the publish-or-perish model of churning out work in favour of a care-based approach.7 The authors draw on histories of feminist theory to critique the way traditionally devalued tasks of ‘care’ and ‘maintenance’—necessary components of any slow movement—were associated with and assigned to feminine labour, while innovation was a masculine endeavour (‘valoriz[ing] production and productivity’) that was recognized and compensated accordingly.8
Maggie Berg and Barbara K. Seeber's The slow professor similarly positions itself as a manifesto in contrast to the growing corporatization of higher education and simultaneous casualization of the labour force within the field; a shift that has, unfortunately, only increased in the years since the book's publication in 2016.9 Berg and Seeber propose slowing down as a matter of ethical import that not only provides individual academics with less stress and a better work–life balance, but—when done at scale—can reduce the unrealistic expectations of productivity that arise when the competition for a dwindling number of jobs becomes more cutthroat each year.
There is an entire sub-field of digital humanities publications dedicated to exposing the invisible labour behind its practices, projects and products.10 Early publications were typically grey literature, such as blog posts, and followed the rise of publications about the #alt-ac movement in the 2010s. This movement, as told by Bethany Nowviskie, began as a way to create solidarity among PhD holders who applied their skills outside the tenure track (‘alt’-ac being more appropriate than the prevailing ‘non’-academic distinction for this type of work), but was quickly co-opted as a ‘solution’ to the academic job crisis and sold to recent PhDs who finished their programmes to discover a profound lack of professional opportunities in academia.11 They were encouraged to simply find ‘alternative’ careers in libraries, museums or the civil service. Publications about invisible labour exposed the way that the people in these roles were often undervalued, underpaid and overworked. The situating of these publications under the banner of digital humanities aligned with the tech boom that offered up technology as another desperate ‘solution’ to a field in crisis—capitalizing on underemployed PhD holders with technical skills and expecting them to provide solutions to problems well beyond the remit of their roles.
The definition used here draws inspiration as well from other modern library theories such as Fobazi Ettarh's ‘vocational awe’; in particular, how technical development is often complicit in perpetuating the idea that something created in support of good intentions must itself inherently be good.12 A ‘slow’ development approach can provide the critical distance needed to determine whether technology will actually solve a problem, or merely delay it until the next funding opportunity comes along. From a technical perspective, the ‘slow’ development approach draws inspiration in part from the creation of Scripto, a tool for crowdsourced transcription that began as a project-specific resource which was then generalized to be used more broadly and which was intentionally created to integrate with commonly used tools for web-based content management, to encourage reuse of existing resources.13
‘Slow’ development in action: the Davy Notebooks Project
In the field of research crowdsourcing, and for digital humanities more broadly, the concepts of novelty and innovation have long been linked to securing grant funding for projects. While understandably driven by the desire to support work that drives research in new and different directions, this has ultimately served to glorify the new rather than reusing or re-investing in existing infrastructure, often to the detriment of the communities served by these resources, who find themselves with an unsustainable, outdated or even broken technical product in a matter of years. Andrew Russell and Lee Vinsel, both founders of The Maintainers research network,14 point out that the ‘devaluation’ of maintenance (both in terms of software and societal infrastructure) goes hand in hand with the ‘fetishization’ of innovation, even though software maintenance often makes up the majority of the effort in the industry.15
Community service is at the heart of the work carried out by maintainers of technical products, but particularly crowdsourcing platforms. This work must respond to the needs of the volunteers providing their labour, as well as the requirements of the researchers creating and running projects. Investing in existing infrastructure benefits communities by providing stability and sustainability; even if familiar tools change, it is easier to adapt to incremental design updates than it is to learn to use an entirely new system. Investing in these tools can lead to quicker response times in support of community needs and requirements, because reusing existing infrastructure often means shorter timelines for work to be completed. This type of shared infrastructure will also support research communities by allowing future efforts to build off existing resources, in particular benefiting under-resourced institutions or those without dedicated research software engineering teams.
This case study will demonstrate how the principles of ‘slow’ development described above were applied in the five years since this collaboration began, and the impacts that following this process has had on the project outcomes. One particular outcome that this case study highlights is that the concept of ‘slow’ development is not indicative of a reduction in development effort. Instead, it is a refocusing of this labour away from novelty, in service of adjustment, testing and iteration—all of which require as much innovation and creativity as the products of novel development.
Collaboration history
The crowdsourcing component of the Davy Notebooks Project began in the summer of 2018, when Professor Sharon Ruston, principal investigator of the project, reached out to Dr Samantha Blickhan to inquire about using the Zooniverse platform. The intention was to secure funding from the Arts and Humanities Research Council (AHRC) in support of a pilot project to transcribe just one of Davy's notebooks, that would then lead to a larger effort to transcribe 44 of his notebooks in total. The notebooks were as yet un-transcribed, and a Davy-focused massive open online course that ran in 2017 had demonstrated that there may be public interest in transcribing Davy's writings.16
Of particular interest to the project team was engaging the public in a way that highlighted the way that Davy's notebooks seamlessly integrate science, poetry, philosophy and art—in stark contrast to their modern separation along the so-called ‘two cultures’ divide.17 At the time, the Zooniverse (a platform that had, until 2017, primarily catered to STEM-focused research efforts like astrophysics and ecology) was in the middle of a multi-year effort to expand their tools in support of humanities-focused crowdsourcing efforts, particularly with respect to tools for text transcription.18 These initial conversations led to a slight increase in scope for the pilot project, mainly due to the average pace of transcription on the Zooniverse platform; in order for the transcription process to last several months (rather than, say, several weeks or even days) there would need to be multiple notebooks transcribed in the pilot phase. This initial conversation led to a multi-year formal partnership between the Zooniverse and the Davy Notebooks team at Lancaster University.
Pilot
The pilot project launched in March 2019, with the aim of crowdsourcing the transcription of just five of Davy's notebooks. The pilot project was created using only the tools available as part of the Zooniverse Project Builder, a browser-based, do-it-yourself tool that allows anyone to create an online crowdsourcing project for free, without requiring any web development skills.19 This means that there was no customization or technical intervention from the Zooniverse team beyond the consultation process described above, as well as the standard technical guidance available to all teams using the Project Builder.
The pilot was supported with AHRC Follow-on-Funding from a leadership fellowship, and the intention was for it to serve as a proof of concept and evidence needed to write a large-scale AHRC research funding bid. Primarily, it was meant to demonstrate that there was indeed interest and excitement from the public not only in Davy's notebooks, but in the act of transcribing them. The project team also intended to use the resulting data from the pilot project to demonstrate what, if any, technical intervention would be necessary to facilitate launching the project at scale. In this way, the results of the pilot would serve to directly justify any request for funding to support technical development, rather than framing the project within the realm of technical novelty.
The pilot project was structured so that each notebook had its own workflow: a series of tasks that a volunteer is asked to do when presented with data in a Zooniverse project classification interface.20 This meant that volunteers would only interact with the pages from one of Davy's notebooks at a time. Each workflow had an identical design, which included a combined annotation and transcription task. Volunteers would use the ‘Text Underline Tool’ by clicking and dragging their computer mouse to create an underline marking beneath a line of written text on the digital image of the page. After they drew the line, a pop-up box would appear, prompting them to transcribe the text they had just underlined (shown in figure 1).
Figure 1. A screenshot from the pilot version of the Davy Notebooks Project (2019) showing the Text Transcription modal and Text Underline Tool.
Volunteers were also given the option of using three different text modifiers, which functioned as wrappers around the text to be modified: ‘unclear’, ‘deletion’ and ‘insertion’. For example, page nine of notebook 13C contains a poem, the first two lines of which include crossed-out and re-written text. In the context of the pilot project, the correct format for volunteers to use in their transcription of those two lines would have been:
To view the heavens with [deletion]solar[/deletion] [insertion]morning[/insertion]
radiance [deletion]white[/deletion] bright.21
Once volunteers had transcribed the text, they would click ‘OK’ and the transcription box would disappear, allowing them to move on to the next line. When they had completed all the lines on the page they wished to transcribe, they were prompted to click ‘Next’, after which they were asked to ‘Tag any sketches on the page by drawing a box around the sketch’ (shown in figure 2). The drawn box would provide positional data in the form of pixel coordinates to the research team about where sketches were located on the digital image. Volunteers could add multiple tags, depending on the number of sketches on the page.
Figure 2. A screenshot from the pilot version of the Davy Notebooks Project (2019) showing the sketch tagging task.
After tagging any sketches on the page (or skipping this task if no sketches were present), volunteers were asked to classify the page according to topic. For the pilot project, these categories were limited to poetry, science, philosophy, geology and ‘other’. Volunteers could select multiple terms, if needed.
In terms of public interest, the pilot was a clear success: volunteers transcribed all five planned notebooks, as well as three additional notebooks, in 75 days. What was striking, however, even beyond the volume of transcriptions completed was the depth of engagement volunteers showed with the subject matter, often following their transcriptions with posts on the project's ‘Talk’ message board to ask questions and share discoveries and other thoughts about the notebooks’ content. Within the first few weeks of the pilot project launch, one volunteer shared this experience of conducting additional research to confirm their transcription of a name:
I was trying to confirm my reading of the river name ‘Awe’, and it looks like it is the river Awe is [sic] Scotland[.] I searched online and found multiple references from the 1800s of geologists mentioning the porphyry there […] including Davy in a Geology lecture[.]22
The post included links to the Wikipedia article for the River Awe, and to a Google Books search result of the Collected works of Sir Humphry Davy, volume 8, published in 1840.23 The immediacy with which the community of transcribers began not only participating in the transcription process but also conducting additional research into the notebooks’ content and sharing it on the project message boards was a clear indication of the opportunity evident through scaling the project to include the rest of Davy's notebooks.
Pilot outcomes, addressing challenges and planning for scale
Between 11 July and 23 September 2019, 505 volunteers transcribed eight of Davy's notebooks. According to a short survey of participants after the pilot, 95% of respondents were at least partially motivated by a desire to contribute to ‘real’ research—specifically selecting ‘To make a contribution to knowledge’ as a response to the question of why they chose to participate in the project. This motivation has consistently held true for the wider Zooniverse community, even across research disciplines.24
Though the pilot was a success in terms of participation and public engagement, the transcription data produced through the pilot project proved difficult to work with. That the resulting data would be complex had been anticipated from the start; concurrent efforts by the Zooniverse team to expand the platform's resources for humanities-focused projects were in no small part focused on creating tools to make it easier for teams to aggregate, review and edit results of text transcription projects.25 While the pilot results were able to be reviewed and made available online, this difficulty translated into one of the main objectives of the second phase: to improve transcription project infrastructure on the Zooniverse platform, not only for use in the Davy Notebooks Project, but for the benefit of future researchers hoping to use Zooniverse to support their own crowdsourced transcription efforts.
The outcomes of the pilot project were incorporated into a proposal for research grant funding from the AHRC, which also marked the beginning of the formal collaboration between Lancaster University and the Zooniverse team. Part of preparing the funding proposal included identifying the technical features that would be retained, what needed to be adapted and what did not yet exist. These are displayed in table 1.
| No. | Problem | Pilot feature description | Proposed change | Requires development? |
|---|---|---|---|---|
| 1 | Volunteers did not like transcribing pages at random | Individual notebook pages presented in random order | Present notebook pages in sequence | Yes, minor |
| 2 | Volunteers said there were too few options for page categorization | Presented volunteers with five options for categorizing pages | Present volunteers with a more comprehensive list of categorization options | No, can change via Project Builder |
| 3 | Sketch tags were too complicated (for both volunteers and researchers) | Asked volunteers to mark sketches by drawing a box on the image | Replace marking task with a Yes/No question to identify pages with sketches present | No, can change via Project Builder |
| 4 | Volunteers noted certain text features were difficult to transcribe, such as equations and tables | Presented volunteers with a single tool for transcribing, and asked them to do their best | Include an option for volunteers to note whether there were elements on the page that were too difficult to transcribe | No, can change via Project Builder |
| 5 | Project Builder did not include all the necessary text modifiers | Presented volunteers with options for ‘insertion’, ‘deletion’ and ‘unclear’ | Add three new Text Modifiers to the Project Builder: ‘underline’, ‘superscript’ and ‘ampersand’ | Yes, minor |
| 6 | Numeric retirement metrics mean there is no way to ensure every page is transcribed completely | Page completion is based on number of people to transcribe the page | Change to using a transcription method that uses a consensus-based approach to image retirement | Minimal, for bug fixes and adjustments only |
| 7 | Raw data is too complex (each page transcribed 3×) | Researchers must identify consensus transcription for each page by reviewing three transcriptions | Use new Zooniverse collaborative editing tool to review and edit transcription results | Minimal, for bug fixes and adjustments only |
There were two categories of problems identified through the pilot project: volunteer-facing issues and researcher-facing issues. As detailed in table 1, there are three categories of response to these issues: those that require development to address; those that do not require development to address; and those that, while they may not require new tools to be developed, are new enough uses of existing features for only a minimal development budget to be necessary, to support team member efforts to fix bugs or make minor adjustments to the code. This choice was based on the Zooniverse team's prior experience launching new features in larger projects and was reflected in the proposed budget for the AHRC grant that supported this work.
Page presentation (1)
A main complaint from volunteers during the pilot project was that it was frustrating to transcribe pages at random, even when each notebook had its own separate workflow. Removing texts from their original context within a given notebook at times led to difficulty determining the transcription of a word, but often the frustration stemmed from volunteers not being able to find out what happened next, or to get a complete sense of the materials they were transcribing.
At the time of the pilot project, there was no dedicated feature on the Zooniverse platform for presenting images in sequential order. The Zooniverse infrastructure was originally designed to present project data at random, to reduce the risk of bias in classification data—one of many examples of how efforts to increase humanities-focused offerings on the platform have had to interrogate the underlying assumptions about how projects should work that were built into the platform from its inception. Over the years, several custom-built projects had included experimental forms of sequential subject delivery, but none had yet produced an approach that worked well enough to make it widely available.
A generalizable approach was determined based on previous project experience, which could be tested at scale via the Davy Notebooks Project. The method requires research teams to include information about subject ordering as part of the image metadata that they upload to the Zooniverse platform along with their project images. On the development side, this work supported the addition of a new, admin-level feature to turn on sequential classification at the workflow level. Ideally, if the method proved successful for the Davy Notebooks Project, it could then be implemented as an option for all project builders, without requiring administrative-level intervention from the Zooniverse technical team.
Page categorization (2)
During the pilot project, volunteers would frequently post on Talk about the page categorization task. They wanted the research team to know that the specific page they were transcribing did not correspond to the limited responses available for the ‘How would you define the notebook page that you have just transcribed?’ question. In their posts, volunteers frequently included suggestions for how they would have categorized a specific page. After some discussion, including a review of Davy's notebooks to be transcribed and their existing catalogues compiled by the Royal Institution, the project team increased the number of categories from five to 13. This change was made using the Project Builder, and therefore required no development effort. As with the pilot, volunteers could select multiple terms, if needed.
Sketch tags and un-transcribable data (3–4)
During the process of reviewing the pilot project data, the team realized that they did not need positional data for the sketches on the page. The positional data produced from the pilot (during which volunteers were asked to draw a box around any sketches on an image) was difficult to aggregate, and the research requirements were met simply by knowing whether a page contained a sketch. Therefore, the team chose to replace the task, rather than replicate something that produced data which was not necessary. Instead of asking volunteers to draw a box around sketches, the task presented volunteers with the question of whether they encountered any sketches on the page.
Another problem noted in the pilot project was that, from time to time, Davy's notebooks included elements that proved particularly difficult to transcribe, like equations and tables. Instead of telling volunteers not to transcribe these difficult components—in fact, a subset of the volunteer community particularly enjoyed transcribing these challenging features—the team chose to provide volunteers with a way to articulate when a page contained material that they did not feel they could accurately transcribe. To do this in a way that did not add another question to the workflow (which would create additional steps/clicks for the volunteers), the team simply expanded the remit of the ‘any sketches?’ question to include any text that would be particularly difficult to transcribe, such as ‘tables of figures, mathematical or chemical equations, or text featuring non-standard keyboard characters’. This provided a way for transcribers who felt less confident with the material to articulate when the contents of a page made it less accessible. On the back end, it was a useful metric for the research team to determine which pages might have an increased need for quality assurance review due to the complexity of the content. This change was made using the Project Builder, and therefore required no development effort.
Text modifiers (5)
During the pilot project, the Project Builder text entry task included the option to provide volunteers with the ‘unclear’, ‘deletion’ and ‘insertion’ text modifiers. The research team also wanted to include three additional options, based on what they learned from the pilot project.
First, they needed an option for ‘superscript’, as Davy frequently used this as a way of writing abbreviations—a common method in his time, although today it is typically only used in handwriting for ordinals. The team wanted to be able to indicate in the final, marked-up transcriptions where Davy had used superscript, and so needed volunteers to be able to tag specific sections of the text in this manner. The second request was for an ‘underline’ modifier, to similarly indicate these places in the final version of the transcriptions. Finally, the team requested an ‘&’ modifier, mainly to indicate to volunteers that this was the character they should use to transcribe Davy's ampersand, rather than the ‘+’ symbol. However, this request was set aside, as the ampersand would need to function as a replacement character rather than as a modifier, causing concern that this difference in behaviour might be confusing for volunteers. Instead, volunteers were given specific instructions on how to transcribe the ampersand in the project Field Guide.26
The ‘superscript’ and ‘underline’ modifiers were added to the Project Builder as part of our custom feature development effort. They are currently available to all users of the Project Builder as Text Modifier options to use with transcription tools.
Page completeness (6)
The approach to text transcription used in the pilot project followed the typical Zooniverse practice of count-based retirement. ‘Retirement’ here is a Zooniverse term for when a particular image has met the requirements for completeness that are set by the project team via the Project Builder. The Zooniverse approach to crowdsourcing is to collect multiple responses for all tasks in a given workflow (in lieu of, for example, having volunteers work asynchronously on a single transcription). Research teams then need to determine the consensus response for each task.
This approach was created to ensure high-quality project results, e.g. so that a project's data would not be skewed if a single volunteer misread the instructions or made a typo. This method was developed in the early days of the Zooniverse platform, when most workflows were made up of short, often question-based, tasks. They were very quick to complete, and there was low risk of the task being submitted when only partially finished. These tasks can be marked as ‘required’ to avoid the risk of volunteers forgetting to answer a question.
The problem arrives when complex task structures—like full-text transcription—are introduced to the platform. Asking volunteers to transcribe entire pages of text requires project teams to assume that everyone who participates in the project is willing and able to contribute a significant amount of time to completing that task. For line-based transcription, it is impossible to ‘require’ a certain number of lines to be transcribed, given that the number of lines on a page will vary across the set of images. This was one of the issues that Zooniverse researchers set out to investigate when beginning to develop new tools for text transcription in 2017; based on the history of transcription projects on the Zooniverse platform, there was evidence that volunteers did not always transcribe an entire page of text in a single classification session.27 One of the results of this effort was a new approach to transcription known as the Transcription Task (shown in figure 3).
Figure 3. A screenshot from phase two of the Davy Notebooks Project (2021) showing the Transcription Task modal, displaying an open dropdown menu containing a single transcription of a line of text provided by a previous transcriber.
When the Davy Notebooks pilot project launched in 2019, the Transcription Task was not yet publicly available; it had been launched on a single custom-built project,28 and was being tested with projects created using the Project Builder. In early project planning conversations, the Transcription Task was identified as potentially being useful for the Davy phase two transcription effort due to several features that aligned with the lessons learned through the pilot project. These included allowing volunteers to see and interact with one another's transcriptions, a greying-out system that indicated when sufficient transcriptions had been received for a given line (helping to spread volunteer effort out across the page) and a retirement metric based in volunteers identifying when all lines on a page had been completed, rather than simply by adding up the number of volunteers who had submitted a transcription for that page.
The Transcription Task did not require direct technical development, as it had already been created and incorporated into the Project Builder by the time that the second phase of the Davy Notebooks Project was ready to launch. However, because it was still a new feature, and the Davy Notebooks Project would be the largest and most complex project to use it, a minimal amount of developer effort was included in the project budget for the implementation of this new task type, in case the work surfaced any bugs or unexpected issues due to the scale of the project.
Data complexity (7)
The resulting data from the pilot project came as JSON embedded in a CSV file. Three transcriptions were collected per image, and so the research team were faced with the task of grouping their transcription data by image, then attempting to reconcile the three transcriptions into a single, authoritative version. Due to the difficulty of attempting to aggregate the three versions into a single transcription, the team took the approach of using the ‘best’ of the three. This is a method that has been used successfully by other Zooniverse transcription project teams in the past, but, unless the process is carried out in a way which makes use of all the transcriptions,29 it can feel ethically dubious in terms of whether it truly justifies asking three people to contribute full-page transcriptions. One of the main tenets of the Zooniverse platform is to not waste volunteers’ time, and, in line with this ethos, the second phase of the project did not replicate this method of independent transcription.
As with the Transcription Task, a new option for working with transcription data was in the process of being developed during the pilot project. The Aggregate Line Inspector and Collaborative Editor, or ALICE,30 was created with funding from the US National Endowment for the Humanities to support teams running text transcription projects on the Zooniverse platform, specifically due to the complexity of working with the Project Builder data output. ALICE was created to be used in conjunction with the Transcription Task and supports automated aggregation of project data. Research teams can view and edit the results of transcription projects in aggregate form, as well as see the individual responses that make up the aggregate version. Once final transcriptions are approved, ALICE data output includes plain text files, to provide a simpler option for teams whose research efforts do not require the positional data and raw data available in the JSON format also included in ALICE exports.
Main development for ALICE concluded in late 2020, and several teams were able to test the app before the Davy Notebooks Project launched its second phase the following year. However, the Davy Notebooks Project was by far the largest transcription effort to launch using the combined Transcription Task/ALICE approach, as well as the first to use sequential transcription. As for the Transcription Task, developer time was included in the project budget in anticipation of any issues that might arise, given that this was the first time so many new features had been combined in a single project, and at this scale.
Phase two beta testing, launch and project maintenance
The beta test for the second phase of the project was held in April 2021. The responses to the changes were overwhelmingly positive and included feedback from new volunteers as well as those who had participated in the pilot project. Some beta testers even posted additional comments on Talk, in which they went into detail about the project redesign, the instructions and their experience using the new tools. Often, volunteers referred to features they had seen in other Zooniverse transcription projects, which they had found useful in their efforts transcribing on those projects. One example reads:
I think it'd be really helpful if there were some samples of [Davy's] letters that may be hard to read similar to the handwriting samples in the “Shakespeare's World” transcription project. I've found what I assume are letter “p”s (only by context […] a word that only makes sense as “deep” looks like it ends with a cursive “b”) and what I think is “ss” but resembles an “f.” I only think those are “ss” due to context and the examples from “Shakespeare's World.”31
In response to this example, the team added additional examples of Davy's writing into the project Field Guide. But this type of feedback provided insight into the community of volunteers taking part in the project. Clearly, participants were engaging in critical palaeography methods and applying the knowledge they had gained in other projects—in this case, familiarity with the long ‘s’ developed through transcribing seventeenth-century English secretary hand in Shakespeare's World.32
The decision to budget additional developer time for bug fixing proved prescient in the lead-up to the project re-launch. While testing the functionality of the new project workflow with all the above adaptations, a number of issues arose due to the integration of so many new or updated features (including one that caused the fairly puzzling display of line annotations shown in figure 4).
Figure 4. A screenshot from phase two of the Davy Notebooks Project (2021) showing an annotation bug that was discovered and fixed before the project's re-launch.
When considering the development effort that went into testing and fixing issues, it becomes clear that the concept of ‘slow’ development is not equivalent to a reduction in development effort. For the Davy Notebooks Project, it was a reallocation of effort away from creating new features, and instead using that time to experiment with using multiple recently created features in tandem. This example is a useful response for those who might argue that novel development is crucial for innovation and creativity—in this experience, the act of combining many existing features into a single project indeed felt like creating something completely new.
The second phase of the Davy Notebooks Project launched on 15 June 2021. Each notebook was presented as a single workflow, so that volunteers could choose which of the available notebooks they wished to work on. Throughout the project, multiple notebooks were available at the same time, to give volunteers a variety of options from which to choose. An example of the project home page, complete with multiple workflow options, is shown in figure 5. After launching, a handful of mid level technical issues surfaced (including one particularly tricky issue with sequential delivery, in which volunteers would, at random intervals, receive the same page multiple times in a row), but were minor enough to not disrupt the project.
Figure 5. A screenshot from phase two of the Davy Notebooks Project (2021) showing the project home page with multiple notebooks to transcribe.
The second phase of transcription was completed on 1 November 2023, just shy of two and a half years after the launch of the second phase. A total of 3415 registered Zooniverse volunteers took part, transcribing more than 10 500 pages of Davy's handwritten text.
‘Slow’ development and ethics; techniques for volunteer engagement
When considering ‘slow’ development as it pertains to ethical practices for crowdsourcing projects, it is also important to consider how the method serves the volunteer community as well as the research and development team(s). The best way to support crowdsourcing communities is transparent communication about the goals of the project, with particular focus on how the contributions made by volunteers will be used in service of those goals. Communication is non-negotiable for ethical crowdsourcing practice; without knowing the full extent of the project, including how their contributions will be used, volunteers cannot make an informed decision about whether they should participate. If there is feature development as part of a project, the instructions should be clearly communicated, and volunteers should have opportunities to provide feedback on the new tools.
For the Davy Notebooks Project, standard communication practice included providing updates on how volunteer feedback impacted the project design changes between the first and second phases, having a consistent presence on the Talk boards, sending regular email updates to project participants, and providing both virtual and face-to-face opportunities like transcribe-a-thons and thank you events, where research team members and volunteers came together to discuss Davy, the process of transcribing, interesting notebook discoveries and how participating in the project changed their understanding of the relationship between the arts and sciences.
Like the ‘careful and care-based approach’ to developing tools for the project, the approach to volunteer engagement could also be described as ‘slow’ in that it has been thoughtful, careful and responsive. Engagement from the project team has been crucial to retaining and supporting the volunteer community, in particular the roughly 20% of volunteers who are the most active transcribers, as well as continuing to grow and reach new volunteers throughout the transcription process. This participation rate is in line with most crowdsourcing communities, who tend to conform to the ‘80/20’ rule, in which 20% of the community contributes 80% of the effort;33 3415 registered Zooniverse volunteers helped to transcribe these notebooks, and a major component of the team's effort was spent facilitating their work.
For those unfamiliar with the process of running a crowdsourced research project, it may come as a surprise to learn how much effort goes into engaging a volunteer community. Yet this was a daily part of the roles of the senior research associate, Dr Andrew Lacey, and Drs Eleanor Bird and Alexis Wolf, research associates on the project who joined in early 2021 (who were also joined for six months by Dr Corrina Readioff). Their main aim was to make volunteers feel valued and to reinforce their contributions to the research efforts, both through transcribing Davy's notebooks and sharing insights via posts on the project's Talk boards. As demonstrated in the pilot phase, the second phase of the project saw a continuation of volunteer-led research into the content of notebooks’ pages, with Talk acting as the central ‘meeting place’ for volunteers and researchers to discuss, debate and share. More than 3000 discussion threads were started over the course of the project, and over 8000 comments posted in the main discussion thread on Talk, each one representing a volunteer or volunteers engaging with the content of a particular page. Many of these contain research carried out by volunteers to identify people, places and books mentioned on the pages, and cross-references to Davy's published works. The research associates have participated in these discussions and checked insights. They were able to use these posts to create a database of over 3000 entries that have fed directly into the annotations in the digital edition of Davy's notebooks.
The team have aimed to reinforce a sense of being part of a community among transcribers by organizing a series of online events where transcribers can meet each other. Transcribe-a-thons have helped the research team to boost the number of transcriptions and reinvigorate enthusiasm for the project, but they have also been occasions for giving back to volunteers by sharing research, providing opportunities to discuss the notebooks with researchers and a chance to meet each other. During the project, the team held several thank you events, organized by Davy research associate Dr Eleanor Bird. At a recorded event for this special issue, volunteers were asked to come and talk about their favourite page or discovery. This event elevated the informal comments that volunteers had made on Talk and formalized them as part of a conference presentation, on a Davy Notebooks Project branded PowerPoint. In this way, transcribers who went above and beyond the contributions of most project volunteers were recognized publicly as members of the research team.
Social media has been another tool to continue to engage participants. Volunteers’ comments in Talk fed into posts by the @davynotebooks Twitter account, and volunteers have mentioned seeing these posts and this leading them to notice pages.34 For the Davy Notebooks Project, Twitter acted as a kind of classroom wall, in terms of displaying and celebrating the achievements of volunteers and researchers as they collaborated on transcribing and researching the content of Davy's notebooks. The ‘slow’ development of the technical components of the Davy Notebooks Project inspired a thoughtful, people-first community environment that fostered placing the needs of volunteers and researchers at the centre of activities. The opportunity for public engagement was evident from the outset by the immediacy with which volunteers during the pilot conducted additional research into the content of pages and shared this with the project team via Talk. The team recognized and fostered this interest and developed it into something new by using social media and events to celebrate and showcase the efforts of the volunteers.
The transcription phase was completed in October 2023, but volunteer engagement did not end there. After the transcriptions were complete, the team created a new Talk board on the project, called ‘Editing’, in which a small community of transcribers continues to participate.35 In this board, the project team posts images of Davy notebook pages, alongside draft versions of the transcriptions provided by volunteers, and Talk participants help to work out any words or phrases that remain unclear.
Final transcriptions of Davy's notebooks, complete with TEI markup provided by the project team, will be published on the Lancaster Digital Collections (LDC) platform in 2024.36 Volunteer transcribers will be acknowledged on the LDC project page via their Zooniverse username, except for those who have chosen to opt out of the acknowledgements list. Zooniverse participants were given notice of this option via email as well as the via the project Talk boards.37
Post-launch technical evaluation and platform-side inspiration
After the project launched, the only user-facing technical changes were minor bug fixes, which were typically discovered by volunteers and shared on Talk. On the project team side, however, there was a consistent process of monitoring the flow of transcription data into the project and adjusting methods to ensure that the resulting transcription data was high-quality (i.e. correct), without resulting in pages being over-transcribed. Over the course of the second transcription phase, the project completeness metrics (i.e. how much agreement between transcribers was required to consider something ‘complete’) were adjusted multiple times, based on analyses of the data being produced, and in line with the shifting needs and abilities of the volunteer community.
At the outset, the completeness metrics for the project were as follows:
| • | Three matching transcriptions for a line of text = the line becomes ‘greyed out’ and no additional transcriptions can be submitted (indicates that that line has reached consensus). | ||||
| • | Three ‘Yes’ responses to the question of whether all the lines on a page are grey = the page is considered complete and is removed from the project. | ||||
In the final year of the project, the community had reached a stage where many transcribers were more familiar with the intricacies of Davy's hand than some of the researchers, and the project team was confident enough in their abilities that the number of matching transcriptions required to turn a line grey was lowered from three to two. This was similarly based on a review of the data coming into the project, weighed against the desire to move the transcription process ahead, to combat the natural decrease in project participation that comes with time. For the final batch of 10 notebooks, only one transcription was required per line, and a single ‘Yes’ response to confirm the page had been completed. The volunteer community had clearly demonstrated their aptitude, and the mutual trust built over time facilitated these decisions. The adjustments were not only made to speed up the process, but out of respect for volunteers’ time; if additional volunteer labour is not necessary to sustain a certain level of data quality, it is the responsibility of a project team to adjust the amount of requisite labour accordingly.
A key component to the ‘slow’ development method is also applying the lessons learned from the current objective to future efforts. Much like the Davy Notebooks Project benefited from previous Zooniverse technical development, the lessons learned through this current effort have already been used to influence other projects, even though they could not be implemented in the Davy project while it was active. This influence has taken place in volunteer-facing feature development, updates to our researcher-facing infrastructure and administrative updates that allow the Zooniverse to better serve user communities.
The main volunteer-facing resource that benefited from the lessons learned through the Davy Notebooks Project is the indexing tool.38 Developed as part of the AHRC-funded Engaging Crowds project,39 the indexing tool gives volunteers greater agency around how they participate in a project by choosing what they want to work on, at the individual subject level. The tool pulls out specific fields from the image metadata and exposes that information to volunteers via an index. They can then use this metadata to choose what to work on based on their own motivations for participating in the project. The indexing tool builds on the workflow-subject set structure of the Davy Notebooks Project as well as the sequential delivery method. The Davy and the Engaging Crowds projects ran concurrently (development on the indexing tool began in 2020 and revisions continued through the end of 2021), and this timing provided the ideal opportunity for the Davy project lessons to feed directly into the indexing tool design and iteration.
The main themes of the Davy lessons that fed into the development and refinement of the indexing tool were: expanding sequential transcription; reducing labour requirements for multi-workflow project setup; and investing in researcher-facing infrastructure.
Expanding sequential transcription
A main piece of feedback from volunteers after the phase two launch was that the option for sequential transcription was an improvement on the pilot, but it was not without drawbacks. There were performance issues with repeat subject delivery (particularly at the start of the project), and it inadvertently degraded user experience in a small, but meaningful, way. For years, whenever Zooniverse volunteers wanted to ‘skip’ the image they received in a project, they would simply refresh the page. The sequential transcription method essentially removes the ability to do this, as the next image in the sequence will not be delivered until you have transcribed the previous one. Similarly, volunteers critiqued the fact that it was difficult to tell where they were within the broader context of the notebook: ‘What page am I currently working on, and how many total pages are there?’
Both critiques were directly addressed in the design of the indexing tool. First, it allows volunteers to page through a dataset, including the images that have been retired. Until this feature was created, images that retired from a project simply disappeared and were no longer accessible by volunteers. Second, the indexing tool features a banner at the top of the image viewer that shows exactly where the image being viewed appears in the context of the wider dataset.
Reducing labour requirements for workflow setup
The Davy Notebooks Project structure gave each notebook its own dedicated workflow. This method of organizing and presenting data worked well for volunteers, but was labour intensive for the project team, who had to set up and configure more than 70 individual workflows over the course of the project. While this ultimately worked for the Davy Notebooks team (with the help of a workflow tracking spreadsheet used to determine when a workflow was ready to launch, when it was completed, etc.), this level of effort would likely be a drawback for other teams who wanted to set up their projects in this way. The main issue is the configuration step: the Transcription Task and ALICE require specific back-end configuration that can only be done by a member of the Zooniverse team, which means that a research team not formally collaborating with the Zooniverse would need to reach out and request this configuration via email each time they wanted to launch a new workflow. This seems trivial, but would be extremely cumbersome for a project using this many workflows; it is also not a sustainable way for the Zooniverse team to provide support to all the research teams using the platform.
This inspired the creation of a feature, again as part of the Engaging Crowds effort, known as Subject Set Selection.40 It allows teams to create a single workflow, to which they can then attach multiple subject sets. When volunteers click on the workflow, they are prompted via a pop-up box to choose a subject set they want to work on, which includes the name of the subject set, as well as its current completeness percentage. Using the Davy Notebooks Project structure as an example, instead of each notebook having its own workflow that volunteers select directly from the home page, volunteers would first click on a single workflow, then choose a notebook from a list of those available to transcribe. This retains the element of choice available to the volunteers, while reducing the amount of labour required in the project setup process.
Investing in administrative infrastructure
A key component of ‘slow’ development is human-centred efforts; it is often easy to forget, from the perspective of the team doing the development, that this work should serve platform maintainers as well. The Zooniverse platform is built and maintained by a very small team of developers, a single designer and research leads who also serve as project managers. Part of the commitment to providing the Project Builder and all its related resources free of charge includes maintaining and sustaining those tools over time. To do that in a way that is respectful of the team size and the limited bandwidth of the humans who make up the team, this method also serves to highlight the ways that development can, and should, be in service of platform maintainers as well as users of the tools being built and maintained.
The lessons of the Davy Notebooks Project led to several administrative features that the Zooniverse project team can now use to configure project-specific features with much less time and effort, including automating the Zooniverse-side workflow configuration process via Python script, so it no longer needs to happen manually. Reducing the labour involved in the process of using new features is critical for their wider reuse and even sustainability, as once automation is set up, the team effort previously required to manually configure features can be reinvested in other tasks. It may seem like a small amount of time to reinvest, but it adds up quickly, particularly when many projects are using features requiring configuration. Wide reuse of a specific feature brings it to the attention of more teams, which can bring opportunities for creative reuse and experimentation that lead to formal partnerships and funding applications, restarting the cycle of reuse and reinvestment described here.
Conclusion: retrospection and ‘slow’ development
When a collaboration like the Davy Notebooks Project comes to an end, it is critical to take the time to reflect on the work and identify what went well, what could have been better and what the team would choose not to repeat. Retrospectives—regular meetings that reflect on the development process—are typical for industry professionals in the technology field, e.g. after each ‘sprint’ for teams using Agile methodologies.41 Longer retrospectives can also be useful at the close of a project for the team to discuss the process in its entirety. This can include successes, failures, technical specifics, communication and collaboration techniques, project management and more.
In the field of academic research, a parallel might be the production of white papers or final reports to funders that detail the process of a project and sometimes include critical reflection on method. This work can often be an afterthought or a box-ticking requirement, rather than an actual tool for values-oriented reflection on collaborative efforts where an entire team is given the opportunity to share their experience. Particularly when work is grant-funded, the end of one project can often mean needing to set one's sights on the future. Tight timelines and unexpected delays can lead to efforts at the end of a project being dropped, and time for reflection is often lost in the chase for the next funding opportunity. The ‘slow’ development method places a lot of value on retrospection, because it supports a definition of success that is not entirely focused on the outcomes of a single project. Practising ‘slow’ development is an investment over time, and, to be effective, teams must share their reflections with others.
Additionally, updates on what is—or is not—being done with the resulting data from a project can provide additional detail for future practitioners around which elements of a project's design should be reproduced. For example, Victoria Van Hyning has produced multiple articles detailing the phases of cleaning and refining the crowdsourced transcription data from Shakespeare's World, which ran on Zooniverse from 2014 to 2018.42 These publications help to provide a realistic view of the effort that is often necessary after a ‘live’ project is completed, and are critical to understanding the overall impact of crowdsourcing projects as well as digital humanities development more broadly.
This case study is an example of a technical approach that has worked for the Davy Notebooks Project, and more broadly for digital humanities development efforts on the Zooniverse crowdsourcing platform. The ‘slow’ development theory is meant to be a guide for others in the crowdsourcing and wider digital humanities fields to identify their own values-driven approach to research collaboration and technical development that centres people over products and promotes reusability and sustainability. It is the hope of the entire Davy Notebooks Project team that sharing these outcomes publicly and discussing their importance and the decision-making processes behind them will raise awareness of the behind-the-scenes effort required to support the deep and impactful research made possible by digital humanities methods, evident in the other articles throughout this special issue.
Acknowledgements
The work described here would not have been possible without the contributions of the Zooniverse volunteer community, as well as those of the Zooniverse web design and development team, including Mark Bouslog, Delilah Clement, Jim O'Donnell, Sean Miller, Shaun A. Noordin, Michelle Yuen and Zach Wolfenbarger.
Development of the Zooniverse.org platform is funded by generous support, including a Global Impact Award from Google, and by a grant from the Alfred P. Sloan Foundation. Funding for the Davy Notebooks Project was provided by the Arts and Humanities Research Council, both for the full project (2021–2024) through the Standard Research Grant scheme, and the pilot project (2019) through the Follow-on Funding for Impact and Engagement scheme.
Data accessibility
This article has no additional data.
Declaration of AI use
We have not used AI-assisted technologies in creating this article.
Footnotes
1 Davy Notebooks Project website, 11 July 2019, https://www.zooniverse.org/projects/humphrydavy/davy-notebooks-project (accessed 18 July 2023).
2 The authors, while all members of the Davy Notebooks Project team, represent Zooniverse platform leadership (Blickhan, Zooniverse co-director) and the research associates who carried out the core labour of Zooniverse project management, community engagement and review of resulting transcription data (Bird, Lacey, Wolf). Therefore, the perspective of this piece is formed from many different contributors to the Davy Notebooks Project, including Davy scholars, project managers, digital humanists and Zooniverse software engineers.
3 Mia Ridge et al., ‘1. Introduction and colophon’, in The collective wisdom handbook: perspectives on crowdsourcing in cultural heritage—community review version [website], 29 April 2021, https://doi.org/10.21428/a5d7554f.1b80974b (accessed 18 July 2023).
4 Julia Glassman, ‘The innovation fetish and slow librarianship: what librarians can learn from the Juicero’, In the library with the lead pipe [web journal], 18 October 2017, http://www.inthelibrarywiththeleadpipe.org/2017/the-innovation-fetish-and-slow-librarianship-what-librarians-can-learn-from-the-juicero/ (accessed 18 July 2023); Carlo Petrini, Slow food nation: why our food should be good, clean, and fair (Rizzoli Ex Libris, 2005).
5 Meredith Farkas, ‘What is slow librarianship?’, Information wants to be free [weblog], 18 October 2021, https://meredith.wolfwater.com/wordpress/2021/10/18/what-is-slow-librarianship/ (accessed 18 July 2023).
6 Dean Spade, Mutual aid: building solidarity during this crisis (and the next) (Verso Books, New York, 2020); Adrienne Maree Brown, Pleasure activism: the politics of feeling good (AK Press, 2019); Jenny Odell, How to do nothing: resisting the attention economy (Melville House Publishing, 2020).
7 Alison Mountz, Anne Bonds, Becky Mansfield, Jenna Loyd, et al., ‘For slow scholarship: a feminist politics of resistance through collective action in the neoliberal university’, ACME: Int. E-J. Crit. Geogr. 14 (4), 1235–1259 (2015), 1238.
9 Maggie Berg and Barbara K. Seeber, The slow professor: challenging the culture of speed in the academy (University of Toronto Press, 2016).
10 Examples include Julia Flanders, ‘Time, labor, and “alternate careers” in digital humanities knowledge work’, in Debates in the digital humanities (ed. Matthew K. Gold) (University of Minnesota Press, Minneapolis, 2012), https://dhdebates.gc.cuny.edu/projects/debates-in-the-digital-humanities (accessed 5 December 2023); ‘Invisible work in digital humanities’ Digital Humanities Quarterly special issue, 13 (2), ed. Tarez Samra Graban, Paul Marty, Allen Romano and Micah Vandegrift (2019), http://www.digitalhumanities.org/dhq/vol/13/2/index.html (accessed 5 December 2023).
11 Bethany Nowviskie, ‘#alt-ac: alternate academic careers for humanities scholars’, Bethany Nowviskie [weblog], 3 January 2010, http://nowviskie.org/2010/alt-ac/ (accessed 5 December 2023).
12 Fobazi Ettarh, ‘Vocational awe and librarianship: the lies we tell ourselves’, In the library with the lead pipe [web journal] 10 January 2018, https://www.inthelibrarywiththeleadpipe.org/2018/vocational-awe/ (accessed 18 July 2023).
13 Sharon Leon, ‘Build, analyse and generalise: community transcription of the Papers of the War Department and the development of Scripto’, in Crowdsourcing our cultural heritage (ed. Mia Ridge), pp. 89–111 (Routledge, London and New York, 2014).
14 The Maintainers [website], https://themaintainers.org/ (accessed 29 February 2024).
15 Andrew Russell and Lee Vinsel, ‘Let's get excited about maintenance!’, New York Times [website], 22 July 2017, https://www.nytimes.com/2017/07/22/opinion/sunday/lets-get-excited-about-maintenance.html (accessed 1 March 2024).
16 ‘Humphry Davy: laughing gas, literature, and the lamp’, FutureLearn [website], 30 October 2017, https://www.futurelearn.com/courses/humphry-davy (accessed 18 July 2023).
18 Victoria Van Hyning et al., ‘Transforming libraries and archives through crowdsourcing’, D-Lib Mag. 23, 5–6 (2017).
19 ‘Build a project’, Zooniverse [website], 29 June 2015, https://www.zooniverse.org/lab (accessed 18 July 2023).
20 ‘Workflow’, Zooniverse glossary [website], 29 June 2015, https://help.zooniverse.org/getting-started/glossary/ (accessed 29 February 2024).
21 Royal Institution, MS HD/13/c, p. 9. View the pilot transcription results at: http://humphrydavy.org.uk/notebooks/note/ri-ms-hd-13-c-p-009/.
22 ‘Davy Notebooks Project Talk’, Davy Notebooks Project [website], 24 July 2019, https://www.zooniverse.org/projects/humphrydavy/davy-notebooks-project/talk/2585/1068067?comment=1754958 (accessed 18 July 2023).
23 The collected works of Sir Humphry Davy, 9 volumes (Smith, Elder & Co. Cornhill, London, 1840), vol. 8, p. 212.
24 Rob Simpson, ‘Who are the Zooniverse community? We asked them …’, Zooniverse [weblog], 5 March 2015, https://blog.zooniverse.org/2015/03/05/who-are-the-zooniverse-community-we-asked-them/ (accessed 5 December 2023); updated confirmation available in Corey B. Jackson et al., ‘The 2021 Zooniverse participant survey’, presented at the 2022 C*Sci conference [conference poster, 23–26 May 2022].
25 Samantha Blickhan, ‘New developments in crowdsourced text transcription’, poster presented at Association for Computers and the Humanities 2021, 21–23 July, https://sites.google.com/zooniverse.org/new-developments-in-ctt/home (accessed 18 July 2023).
26 In the context of Zooniverse projects, a ‘Field Guide’ is a pop-up area of a project where teams can include content-specific examples and advice, including things like handwriting samples or lists of common abbreviations.
27 Samantha Blickhan et al., ‘Individual versus collaborative methods of crowdsourced transcription’, J. Data Mining Dig. Hum. special issue, ‘Collecting, preserving, and disseminating endangered cultural heritage for new understandings through multilingual approaches’ (2019), https://dx.doi.org/10.46298/jdmdh.5759 (accessed 18 July 2023), pp. 8–9.
28 Anti-Slavery Manuscripts at the Boston Public Library [website], 23 January 2018, https://www.antislaverymanuscripts.org (accessed 5 December 2023).
29 Nathaniel Deines et al., ‘Six lessons from our first crowdsourcing project in the digital humanities’, Getty [weblog], 7 February 2018, https://www.getty.edu/news/six-lessons-learned-from-our-first-crowdsourcing-project-in-the-digital-humanities/ (accessed 18 July 2023).
30 ALICE: The Aggregate Line Inspector and Collaborative Editor [website], 2020, https://alice.zooniverse.org/about (accessed 5 December 2023).
31 Op. cit. (note 22), https://www.zooniverse.org/projects/humphrydavy/davy-notebooks-project/talk/2586/1055340?comment=1736497 (accessed 5 December 2023).
32 Shakespeare's World [website], 8 December 2015, https://www.shakespearesworld.org (original site now archived; current site accessed 5 December 2023).
33 Alexandra Eveleigh et al., ‘Designing for dabblers and deterring drop-outs in citizen science’, Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, Apr 26–May 1, Toronto, ON, pp. 2985–2994 (2014), https://doi.org/10.1145/2556288.2557262.
34 https://twitter.com/davynotebooks; Twitter has since rebranded into ‘X’, though the Twitter-branded URLs persist.
35 ‘Editing Talk Board’, Davy Notebooks Project [website], 14 July 2019, https://www.zooniverse.org/projects/humphrydavy/davy-notebooks-project/talk/6122 (accessed 5 December 2023).
36 Lancaster Digital Collections [website], https://digitalcollections.lancaster.ac.uk/ (accessed 27 February 2024).
37 Op. cit. (note 22), 15 January 2024, https://www.zooniverse.org/projects/humphrydavy/davy-notebooks-project/talk/2644/3229678?comment=5295814 (accessed 1 March 2024).
38 Samantha Blickhan, ‘Engaging Crowds: new options for subject delivery & interaction’, Zooniverse [weblog], 3 November 2021, https://blog.zooniverse.org/2021/11/03/engaging-crowds-new-options-for-Subject-delivery-interaction/ (accessed 18 July 2023).
39 Louise Seaward et al., ‘Engaging Crowds: citizen research and cultural heritage data at scale’, Zenodo [web resource], 6 October 2022, https://doi.org/10.5281/zenodo.7152031 (accessed 18 July 2023).
41 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, et al., Manifesto for Agile Software Development [website] 2001, https://agilemanifesto.org/ (accessed 28 February 2024).
42 Victoria Van Hyning, ‘Harnessing crowdsourcing for scholarly and GLAM purposes’, Lit. Compass 16, 3–4 (2019), https://doi.org/10.1111/lic3.12507 (accessed 12 April 2024); Victoria Van Hyning and Mason A. Jones, ‘Data's destinations: three case studies in crowdsourced transcription data management and dissemination’, Startwords 2, Scribes (2021). http://doi.org/10.5281/zenodo.5750691 (accessed 12 April 2024).
