Cyberling Wiki - Recently Updated Pages

Group 2: Standards for Storage, Retrieval, and Search of Data

AliciaBW — Fri, 09 Oct 2009 13:57:06 CDT

As part of the Cyberling2009 workshop at Berkeley, this working group was charged with identifying and documenting existing and needed standards for the digital storage, retrieval, and search of linguistic data. Another concern was the potential for reuse of language data by parties other than the original creators of the data. We present the results of our working sessions as a set of wiki pages, as outlined below.

Members ---
Debbie Anderson, Eric Kansa, Pavel Mihaylov, Johanna Nichols, Alexis Palmer, Alicia Wassink

Process ---
While standards of some kinds for storage, retrieval, and search of linguistic data do exist in linguistics, many subfields of linguistics talk more about "best practices" and "common practices" than they do about "standards". We discussed the ways these terms are used on our Big Ideas page. We were not tasked with discussion of the related and important issue of annotation standards. For discussion of this issue, please see Group 1: Annotation Standards.

Summary of Big Ideas regarding data sharing (storage, retrieval and search):
How do we define 'other standards'?
Standards vs. Best Practices
How do we encourage adoption of standards in linguistics?
Data sharing: the publication model (for more on this, see the white paper from Group 4)
Standards are great, now how do I use them?
How can I participate in the creation of ISO standards?

Issues addressed within the WG2 wiki pages:
Unicode character encoding standards for increasing stable display, readability, and sharing of data
Relational database storage
Wiki-based sharing of research
Metadata tags for increased transparency and usability of data
Version control
Web standards for sharing datasets
Machine reusability of data (under construction)

Results ---

A set of examples/case studies demonstrating applications of standards for storage, retrieval, and search and their utility for linguistic research.
A set of subfield-specific seed lists of common practices, requirements, conventions, etc. The purpose of creating these lists is twofold. First, the lists should be helpful to individual linguists working in the subfield in question. Second, they can serve as reference material to linguists from other areas who might wish to annotate beyond their individual research concern.
A seed list of existing standards for storage, retrieval, and search of linguistic data.
A handful of recommendations regarding not-yet-existent but needed standards for linguistics cyberinfrastructure.
Additional resources: relevant links, papers, etc.

Notes from working sessions

Member Bios

TerryLangendoen — Sat, 26 Sep 2009 17:33:13 CDT

Working Group 1: Annotation Standards

Mary Beckman (co-chair)

wiki username: mebeckman
website: http://ling.osu.edu/~mbeckman
email: mbeckman@ling.osu.edu
I am a linguist who has worked (among many other things) on developing a framework for building annotation conventions for prosodic categories in specific language varieties (see http://ling.osu.edu/~tobi) and on developing a database of child and adult productions of word forms that target lingual obstruents (see http://ling.osu.edu/~edwards). Some of my current funded work is described at http://www.ling.ohio-state.edu/~edwards/socialDynamics/NSFhighlight.html and I am interested (among other things) in developing the infrastructure for facilitating this kind of collaborative approach that incorporates corpus-based work, experimental work, and computational modeling.

Stuart Robinson (co-chair)

wiki username: stuartrobinson
website: https://www.zapata.org/stuart
email: stuart@zapata.org

Sarah Churng
wiki username: ashragurnch
website: http://students.washington.edu/ashra/
email: ashra@u.washington.edu
I am a graduate student at the University of Washington. My main goal at the Cyberling workshop is to foster discussion for how cyberlinguistic infrastructure may facilitate sign language transcription and annotation, as part of the Annotation Standards working group, through best practice efforts, dissemination through the web, etc. I hope to compound these discussions with current research in deaf literacy, in which the first language acquisition of sign languages is hypothesized to bootstrap spoken language literacy.

Greville Corbett

wiki username:
website: http://www.surrey.ac.uk/LIS/SMG/gcorbett.htm
email:
I work at the University of Surrey, where I lead the Surrey Morphology Group. I have worked particularly on the typology of features, as in Gender (1991), Number (2000) and Agreement (2006), which makes me very positive towards the Leipzig Glossing Rules, while aware of what still needs to be done there. The SMG has produced several typological databases, freely available over the web (http://www.surrey.ac.uk/LIS/SMG/web_resources.htm ), so I have a particular interest in how we ensure that the research and effort involved in setting up such resources is respected and acknowledged. I collaborated with Marina Chumakina, Dunstan Brown and Harley Quilliam on the Archi Dictionary (http://www.smg.surrey.ac.uk/archi/linguists/index.aspx) , with all the issues of scripts, sound files, picture files, web access and so on.

Chuck Fillmore

wiki username: cjfillmore
website:
email: fillmore@icsi.berkeley.edu
A long-retired member of the Berkeley linguistics department. Associated since retirement with the "FrameNet" project, producing a kind of valency dictionary whose categories are based on knowledge of the semantic frames that underlie individual words' meanings (http://framenet.icsi.berkeley.edu). The analyses are based on annotations of sentences extracted from a large corpus of written English. Currently also trying to use the same methods to build a registry of "minority grammatical constructions" (the ones that ordinary parsers can't handle, or that familiar compositional principles can't interpret). I've been trying with some student colleagues to develop an abbreviated way to annotate phrases in respect to how they instantiate individual constructions. Each construct annotation has to be keyed to a full description of the construction, in the way that each lexical annotation has to be keyed to a full description of the relevant frame. Annotation of all the words or all the constructions in individual sentences requires layered stand-off annotation sets.
I lacked the patience to complete the complicated interview that would have allowed me to insert a photo - I could have tried to find out what my zodiac sign is, but I don't have a good photo anyway: I'm tallish, red-haired, slow-moving, walk with a cane.

Richard Wright

wiki username: rawright@u.washington.edu
website: http://depts.washington.edu/phonlab/people/wright.html
email: rawright@u.washington.edu
I am an associate professor of linguistics and the Phonetics Lab director at the University of Washington. I specialize in phonetic research and field lingiustics primarily in the acoustic domain. My interest in the Cyberling workshop stems from my experience developing and working with corpora of spoken language (in part with Alicia Wassink), in working on voice-based human machine interfaces ("The Vocal Joystick" ), and in working with Steve Moran) to develop a typological database of phonological inventories tied to ontological models of phonological feature theories that expands on previous work such as UPSID and the Stanford Phonology Archive (P HOIBLE). I am particularly interested in methods for representing the sounds of languages in ways that are machine readable and standardized across languages and across applications.

Working Group 2: Other Standards

Johanna Nichols (co-chair)

wiki username: Johanna.Nichols
website: http://linguistics.berkeley.edu/~ingush/, http://socrates.berkeley.edu/~jbn/
email: johanna@berkeley.edu

I am Professor emeritus in Slavic linguistics at UC Berkeley. I co-founded and co-direct (with Balthasar Bickel; see below) the Autotyp databases and research project (http://uni-leipzig.de/~autotyp/), which will probably have its complete genealogical classification on-line by the time the Cyberling workshop begins. I work on questions of phylogeny, detecting and demonstrating linguistic relatedness, deep linguistic prehistory, and typology, and I have a documentation project creating very large corpora of spoken Ingush and Chechen (East Caucasian). All of these projects require combining data from different fields (linguistics, archaeology, ethnography, human genetics). I am concerned with seeing standards and tools developed that will not require every documentary linguist to start from scratch electronically, and that will serve linguists and language users rather than vice versa. I have developed all-lower-ascii practical writing systems for Ingush and Chechen and a similar system for my East Caucasian etymological database and am distressed at what I perceive as increasing pressure to put corpora, syntactic examples, dictionary headwords, etc. into unreadable phonetic transcription just because of font availability. For my Ingush corpus I'm working out ways of interlinearizing and lemmatizing in languages where lemmas, inflectional categories, etc. are properties of clauses rather than words. I'm also concerned with the lack of instructions, documentation, labels, etc., etc. in languages readable by speakers of minority languages in Russia.

Alexis Palmer, Saarland University (CoLi) and the University of Texas at Austin (co-chair)

wiki username: alexispalmer
website: http://comp.ling.utexas.edu/apalmer
email: apalmer@coli.uni-sb.de
In May 2009 I began a new position as a postdoc in the M2CI Cluster of Excellence at Saarland University in Saarbrücken, Germany. There I am working with Caroline Sporleder in a research group on Computational Modelling of Discourse and Semantics. I'm also just finishing my PhD in computational linguistics at the University of Texas at Austin, under the supervision of Jason Baldridge and Katrin Erk. My thesis research, which is connected with the EARL project, has to do with integrating automatic labeling and human annotation for more efficient production of interlinear glossed text (IGT). In general, I'm interested in the potential for applying techniques and methodologies from computational linguistics (CL) for research in other linguistic subfields, and particularly for documentation of endangered languages. Bridging the space between CL and the rest of linguistics raises a host of issues -- data formatting and availability, code reusability and availability, modularity and generalizibility of computational models, etc. -- related to the aims and core ideas of Cyberling 2009.

Debbie Anderson

wiki username: DeborahAnderson
website: http://linguistics.berkeley.edu/sei/
email: dwanders@sonic.net
I run a project at UC Berkeley, the Script Encoding Initiative, that assists groups (and individuals) in getting eligible scripts and characters into the Unicode Standard/ISO 10646, the international character code standard. I am particularly keen on being sure linguists and members of the user communities (especially minority language speakers) get a voice in the development of standards. I am the UC Berkeley representative to the Unicode Consortium, and a member of the US delegation to ISO/IEC JTC 1 SC2 Working Group 2 on coded character sets.

Eric Kansa

wiki username: ekansa
website: http://isd.ischool.berkely.edu
email: ekansa@ischool.berkeley.edu
Eric C. Kansa is Executive Director of the Information and Service Design Program and is an Adjunct Professor at the UC Berkley School of Information (I School). His primary role is to develop service design projects that bring I School students and faculty to work in collaboration with partner organizations. His research interests include efforts to enhance the accessibility and usability of research data collected in the field sciences, as well as, the impact of ubiquitous information accessibility in the consumer experience of services. Before coming to UC Berkeley, Eric was cofounder and former Executive Director, a nonprofit organization, the Alexandria Archive Institute. There he led development ofOpen Context, an online system for publishing primary research data collected in the field sciences. This follows a position on the faculty of Harvard University, where he served as Lecturer and Undergraduate Tutor for the Department of Anthropology. He graduated from the University of California, San Diego with a BA in Cultural Anthropology. Eric was awarded a doctorate in Anthropology at Harvard University in 2001. Eric is currently Convener of the Society for American Archaeology's Digital Data Interest Group.

Pavel Mihaylov

wiki username: pavel.mihaylov
email: bin,at,bash,dot,info
I am a computational linguist working for Ontotext, a mixed industry/research company based in Sofia, Bulgaria. My main occupation is web mining/information extraction and finite-state morphologies. Together with Dorothee Beermann, I work on TypeCraft (see Dorothee's bio). Other than the computational bit, I have a general interest in linguistics and languages.

Alicia Wassink

wiki username: AliciaBW
website: http://faculty.washington.edu/wassink/
email: wassink@u.washington.edu
I work in acoustic phonetics, sociolinguistics and creole linguistics. In the Cyberling workshop, I'm wearing my sociophonetician hat. I am currently co-authoring a chapter on best practices in sociophonetics pertaining to the instrumental analysis of vowels for addressing research questions of interest to sociolinguists. This chapter is about setting standards vis-a-vis best practices in data analysis, and isn't so much about data storage, retrieval or archiving, which are all topics I'm interested in discussing as part of this working group. As director of the Sociolinguistics Laboratory at the University of Washington, Dept. of Linguistics, I've run metadata tutorials to train my students to use established protocols for associating metadata with their audiofiles that will increase data accessibility, searchability, acoustic analysis and ease of querying.

Working Group 3: Tools

Bill Byrne (co-chair)

wiki username: bill_byrne
website: About me
email: billb@google.com
I design speech user interfaces at Google (e.g. iPhone app) and have been in this field for the last ten years. Having completed a linguistics PhD in 1998, I continue to follow theoretical and applied work but I have always been troubled by the lack of data available to researchers. I see Cyberling as an interesting and encouraging activity for the field. I would love to help develop more ways for linguists in all subdisciplines to easily gain access to very large sets of data as well as share their own data with the rest of the world.

Robert Forkel (co-chair)

wiki username: robert_forkel
website: https://dev.livingreviews.org/projects/epubtk/wiki/people/robert
email: forkel@mpdl.mpg.de
I am a mathematician-turned-software developer working at the Max Planck Digital Library. My interest for web infrastructure for linguistics started while working on WALS Online. Having a couple more projects coming in, I'm interested in ways to publish linguistic data as part of the Linked Open Data Cloud. The data I'm concerned with mainly is word lists, interlinear glosses, etc.

What I hope to take away from cyberling is a clearer idea about the lowest level of quality/granularity which would make sharing data still fruitful - I'm looking for low-hanging fruit.

Dorothee Beermann

wiki username: DorotheeBeermann
website: http://www.hf.ntnu.dorothee.beermann
email: dorothee.beermann@hf.ntnu.no
I am associate professor at the Norwegian University of Science and Technology (NTNU) in Trondheim, Norway.
My interest in Cyberlinguistics results from working across linguistic fields, from grammar engineering on the one hand to African linguistics on the other. Perhaps in particular when working across frameworks one would like to know what defines linguistics as a whole. Can we for example find a common answer to the question: 'What defines linguistic methodology?'?
I am in particular interested in the role that interlinear glosses (IGs) play in linguistic research. Not so happy with the role IGs play at present, I would like to help facilitate a development where they become an independent linguistic resource - accessible to all of us.
Together with Pavel Mihaylov I have created a linguistic tool that helps to generate, store and retrieve IGs in a setting that allows sharing them with a group of colleagues or to publish them online. (TypeCraft).

Arienne Dwyer

wiki username:
website:
email:
[insert bio here]

Florian Jaeger, University of Rochester

wiki username: FlorianJaeger
website: http://www.hlp.rochester.edu [nevermind the security warning; trust me]
email: fjaeger@bcs.rochester.edu
I received my M.A. in Linguistics and Computer Science (HU & TU Berlin, with a visit to UC Berkeley) and my PhD in Linguistics with a designation in cognitive science (Stanford University with a visit to MIT). I have been at Rochester (Brain and Cognitive Sciences and Computer Science) since 01/2007, where I working on efficient language production, maintenance of probabilistic linguistic representations, and other such stuff. My involvement in Cyberling 2009 is related to my interests in replicability and extensibility of scientific work. This implies development of tools and annotation standards that make the data sets developed by one researchers useful to others. I am also interested in cheap `technology' with possible high impact factor, such as taking laptops to the field to run psycholinguistic studies, or the use of online platforms like Mechanical Turk to elicit large amounts of data from many languages at low cost. This often results in unbalanced, highly clustered data (similar to corpus data) for which modern statistical methods are required (in which I am also interested ... conveniently).

Jeremy G. Kahn

wiki username: JeremyKahn
website:
email: jgk@washington.edu
I am a Ph.D. student in Linguistics at the University of Washington. I work as a Research Assistant to Mari Ostendorf in the Signal, Speech and Language Interpretation laboratory within UW Electrical Engineering, and I am currently a Visiting Fellow at the SRI Speech Technology and Research laboratory in Menlo Park. My primary research area is in using syntactic information to support a speech-recognition and machine-translation pipeline. I am interested in data annotation both on principle and as a matter of necessity; heavily statistical research software systems have tremendous issues with data compatibility, portability, and reconciliation.

Virach Sornlertlamvanich

wiki username: virach
website: http://www.tcllab.org/virach
email: virach@tcllab.org
I am the Assistant Executive Director of National Electronics and Computer Technology Center (NECTEC), and the Co-director of Thai Computational Linguistics Laboratory (TCL). My research interests lie in the computational linguistics that covers morphological, syntactic and semantic representation and analysis. My previous works have been provided in publications and implementations such as LEXiTRON (English-Thai dictionary), Asian WordNet, Kui (Knowledge Unifying Initiator: online collaboration editing tool), word segmentation (SWATH), ORCHID (Thai POS tagged corpus), ParSit (English-Thai online machine translation), Sansarn (Thai search engine portal), etc. Currently, I am working as a chair of Asian Language Resource group of AFNLP (Asian Federation of NLP), director member of AAMT (Asia-Pacific Association for Machine Translation). I am also conducting a series of school of ADD (Asian Applied NLP for linguistics diversity and language resource Development) for NLP networking and collaboration in language resource development especially for Asian countries.

Working Group 4: Data Reliability & Provenance

Peter Austin (co-chair)

wiki username: pkaustin
website: http://www.hrelp.org/aboutus/staff/index.php?cd=pa
email: pa2@soas.ac.uk

I am Marit Rausing Chair in Field Linguistics at the School of Oriental and African Studies in London and Director of the Endangered Languages Academic Programme. My research interests lie in the theory and practice of language documentation and description, endangered languages, morphosyntactic typology, Lexical Functional Grammar, and languages of eastern Indonesia and Aboriginal Australia. At SOAS I teach a course on Technology and Language Documentation that covers data modeling, workflow, metadata, archiving, ethics and protocols, and software tools, and have participated in training workshops that cover these topics.

Martin Haspelmath, Max Planck Institute for Evolutionary Anthropology

wiki username: haspelmath
website: http://www.eva.mpg.de/lingua/staff/haspelmath/home.php
email: haspelmath@eva.mpg.de

I am a typologist interested in linking structural data from as many different languages as possible (typological databases). As of 2008, the World Atlas of Language Structures (http://wals.info) has been online, and I'm interested in getting more such datasets out. I hope that linguists will soon publish their dictionaries and corpora online, and that it will become normal to see such online resources as regular (peer-reviewed) publications, because without the incentive of regular publication I fear linguists will not share their materials.

Kurt Bollacker

wiki username:
website:
email:
[insert bio here]

Tracy Holloway King

wiki username: tracyhollowayking
website: http://www-csli.stanford.edu/~thking/
email: tracyhollowayking@gmail.com
I am on the LSA TAC committee (with a number of people on this list). I currently manage the natural language engineering groups at Powerset, a semantic search company acquired in 2008 by Microsoft. I have been a co-organizer for the first three grammar engineering across frameworks (GEAF) workshops. I am particularly interested in making sure that resources, platforms, and theories created can be used cross-linguistically.

Koenraad de Smedt

wiki username: Koenraad
website: http://ling.uib.no/desmedt/
email: desmedt@uib.no
I am professor of Computational Linguistics at the University of Bergen, Norway and head of the Research Group on Language Models and Resources (LaMoRe). Lately I have been working on parsebanking. I am currently the national contact person for CLARIN in Norway and a member of the Science Opportunities Panel of the eVITA Programme Committee (Research Council of Norway). I am also coordinator of the CLARA, a Marie Curie ITN which will start up in the fall of 2009 (see Organizations and Initiatives). In 2007 I organized a Workshop on Unified Linguistic Annotation.

Paul Trilsbeek

wiki username: paul.trilsbeek
website: http://www.mpi.nl/people/trilsbeek-paul
email: Paul.Trilsbeek@mpi.nl
I am an archive manager for the language archive at the Max Planck Institute for Psycholinguistics, part of which is the DOBES archive of endangered languages. At MPI we also develop an array of linguistic tools and a framework for digital archiving of language resources. In the European CLARIN project, which aims at creating a common infrastructure for language resources and technology, the MPI plays an important role in the technical work package (WP2).

Working Group 5: Models from Other Fields

Scott Farrar (co-chair)

wiki username: sofarrar
website: http://faculty.washington.edu/farrar/
email: farrar@u.washington.edu
I am an Assistant Professor of Linguistics at the University of Washington, teaching computational linguistics in the Professional Master's in Computational Linguistics Program at the University of Washington. My primary interest is in computational linguistics with a focus on e-linguistics, or how to apply computational techniques in traditional linguistics research. I received my PhD in Linguistics from the University of Arizona in 2003. Before joining the CLMA Program, I worked at the University of Bremen in Germany and in Cameroon on a fieldwork assignment researching endangered Beboid languages. I am currently funded by the National Science Foundation on grant BCS-0720670 entitled, "Implementing the GOLD Community of Practice: Laying the Foundations for a Linguistics Cyberinfrastructure."

Terry Langendoen (co-chair)

wiki username: TerryLangendoen
website: http://linguistics.arizona.edu/~langendoen
email: langendt@email.arizona.edu
I am Professor Emeritus of Linguistics at the University of Arizona, having retired from academia in 2005. From 2006 to 2008, I was a Program Director in Linguistics at the National Science Foundation, and for the past year have been working part-time as an Expert in the Robust Intelligence Program in the Division of Information and Intelligent Systems at NSF. I thank Nancy Ide for getting me interested in the problem of annotation of electronic linguistic data by inviting me to became part of the NEH-supported Text Encoding Initiative (TEI) in 1987. In that project, I worked with Gary Simons to develop recommendations for the encoding of linguistic structure in SGML, the predecessor to XML, including a general-purpose annotation format for feature structures. In 2001, I began work on the NSF-supported E-MELD project, and together with Scott Farrar and Will Lewis developed the initial specifications of the General Ontology for Linguistic Description (GOLD), with the idea of enabling linguists to annotate their data without being committed to specific markup syntax, as in TEI, and to enable data annotated in different ways to be computationally interoperable. At the January 2009 LSA Annual Meeting, Emily Bender and I organized a special session on Computational Linguistics in Support of Linguistic Theory; in our presentation we touched on several of the themes of this workshop. I discussed Cyberling 2009 briefly at the end of my contribution "Opportunities at NSF" in the special section entitled "Keeing Science Moving in Tight Times" in the September 2009 issue of the Association for Psychological Science Observer.

Balthasar Bickel

wiki username:
website:
email:
[insert bio here]

Steve Moran

wiki username: Steve_Moran
website: http://staff.washington.edu/stiv/
email: stiv@u.washington.edu
I am a PhD student in the Linguistics Department at the University of Washington. My research interests including language documentation and developing cyberinfrastructure for interoperability of linguistic data. As a field linguist I am active in Prof. Jeffrey Heath's Dogon languages project that is documenting the Dogon languages of Mali, and creating an online comparative lexicographic (and multimedia) website (http://dogonlanguages.org/). With Richard Wright, I am also developing a typological database of phonological inventories that is tied to ontological models of phonological feature theories. We are making these resources available at our project website, PHOIBLE (http://phoible.org/). Previously I worked for the Linguist List, specifically on the E-MELD project to foster the consensus of "best practice" standards for the digital archiving of endangered languages data.

Cornelius Puschmann, University of Düsseldorf

wiki username: coffee001
website: http://ynada.com/
email: cornelius.puschmann@uni-duesseldorf.de
I am a postdoc at the Department of English Language and Linguistics at the University of Düsseldorf, Germany. My involvement in Cyberling 2009 stems from my role as the technical coordinator of eLanguage, the LSA's Open Access publishing platform. I am also a strong proponent for Open Access and Open Data in linguistics and in other disciplines.

Dwight van Tuyl, Eastern Michigan University

wiki username: dvantuyl
website: http://linguistlist.org/people/dwight.html
email: dwight@linguistlist.org
I'm a programmer at the LINGUIST List at Eastern Michigan University. We've recently finished the GOLD Community website at http://linguistics-ontology.org which attempts to build a community around the General Ontology of Linguistic Description currently being developed by Scott Farrar of the University of Washington. At the LINGUIST List, we plan on using GOLD in our latest project, LEGO, for annotating lexical data with GOLD concept URI's. I'm hoping to come back from this workshop with an understanding of what tools andinterpolatablestandards could be used for projects like LEGO in order to provide a low barrier of entry for participating in a cyberinfrastructure for linguists.

Working Group 6: Funding Models

Mark Liberman (co-chair)

wiki username: MarkYLiberman
website: http://ling.upenn.edu/~myl
email: MarkYLiberman@gmail.com
Professor (Linguistics, Computer and Information Sciences) at University of Pennsylvania; Director, Linguistic Data Consortium.

David Lightfoot (co-chair)

wiki username: DavidLightfoot
website:
email: lightd@georgetown.edu
David Lightfoot writes mainly on syntactic theory, language acquisition and historical change, which he views as intimately related. He argues that internal language change is contingent and fluky, takes place in a sequence of bursts, and is best viewed as the cumulative effect of changes in individual grammars, where a grammar is a "language organ" represented in a person's mind/brain and embodying his/her language faculty. That, in turn, entails a non-standard view of language acquisition as "cue-based." He has published eleven books, most recently The Development of Language (Blackwell, 1999), Syntactic Effects of Morphological Change (ed.) (Oxford UP, 2002), The Language Organ (with S.R. Anderson) (Cambridge UP, 2002), and How New Languages Emerge (Cambridge UP, 2006). He is also the author of more than 100 articles, book chapters and reviews. He is general editor for the Generative Syntax series published by Blackwell, and serves on the linguistics editorial board at Cambridge University Press. In 2004, he was elected a fellow of the American Association for the Advancement of Science, and in 2006, as a fellow of the Linguistic Society of America.

Dr. Lightfoot has held regular professorial appointments at several universities including McGill University, where he taught many undergraduates who went on to become major figures in linguistics and psychology including Mark Baltin, Alan Prince, Michael Rochemont, Alison Gopnik, Elan Dresher, Norbert Hornstein, Amy Weinberg, Renée Baillargeon and Elizabeth Cowper; the University of Utrecht in the Netherlands; and the University of Maryland, where he established and chaired for 12 years, a new department of linguistics with a unique focus--viewing linguistics as the study of the human language organ. He was also the associate director of the neuroscience and cognitive sciences program there. In 2001, he moved to Georgetown University as dean of the graduate school. In addition, he has held short-term appointments at universities in Austria, Brazil, Canada, Germany, Switzerland and the United Kingdom. In June 2005, he became assistant director of the National Science Foundation, heading the Directorate for Social, Behavioral and Economic Sciences.

Anthony Aristar

wiki username: aristar
website: http://linguistlist.org/aristar
email: aristar@linguistlist.org
I started life as a historical linguist and typologist, but was drawn, during the early Internet age (it was called Arpanet then!) to its potential as a medium for academic exchange. I founded the LINGUIST List in 1990 as a step in this direction, and the list grew so much that I and my co-Moderator Helen Aristar-Dry realized that we needed to rethink what we were and where LINGUIST could go. We started started applying for grants to build infrastructure for the discipline, and in 2004 Helen and I became co-Directors of the Institude for Language Information and Technology, housed at Eastern Michigan University. Our focus for the last few years has continued to be linguistic infrastructure, but we also now deal extensively with standards for linguistics on the Internet (e.g. our EMELD project) and work on digitizing endangered languages data. We have the following ongoing projects, funded by either NSF or NEH: MultiTree (http://multitree.linguistlist.org/) which is collecting all known hypotheses on language relationships, LLMAP (http://llmap.org) which is aimed at making GIS a fundamental part of linguistics, GOLDComm (http://linguistics-ontology.org/), which is aimed at expanding the GOLD ontology for linguistic description, LEGO (http://linguistlist.org/projects/lego.cfm) which has as its goal the development of several "building blocks" for lexical data interoperability within linguistics, and RELISH, a collaborative project with the Max Planck Institute for Psycholinguistics and The Johann Wolfgang Goethe-Universität Frankfurt, aimed a unifying two digital collections of endangered languages with special attention given to harmonizing the European and American standards for language documentation and lexicon building.

Collin Baker

wiki username: collinfb
website: http://framenet.icsi.berkeley.edu
email: collinb@icsi.berkeley.edu

I am a linguist, working as manager of the FrameNet Project, founded and directed by Prof. Charles Fillmore, which is part of the AI group at the International Computer Science Institute in Berkeley. For the last decade, we have been building a rich lexical semantic database for English, based on frame semantic principles and grounded on manually annotated corpus examples of usage. We are currently participating in a joint annotation project for the American National Corpus (http://americannationalcorpus.org), collaborating with colleagues building FrameNets for Spanish, German, Chinese, Japanese, etc., planning an alignment of FrameNet with WordNet, and exploring crowdsourcing as a means of gathering annotation data. I am interested in the problem of funding resource building, particularly long-term efforts.

Helen Dry

wiki username:
website:
email:
[insert bio here]

Laura Welcher

wiki username: lbwelch
website: http://www.rosettaproject.org
email: laura@longnow.org

I direct The Rosetta Project at The Long Now Foundation in San Francisco, and am one of the co-organizers of Cyberling 2009. My interest in cyberlinguistics originally developed out of my experience in linguistic fieldwork, using specialized tools like Shoebox/Toolbox, as well as trying to make general tools like Filemaker Pro work for lexicography. Both of these tasks quickly gave me the sense that better tools are needed for what linguists do! Besides language documentation, my work at The Rosetta Project has underscored the need for standards upon which to build tools. The Rosetta Project maintains an archival collection for all of the world's languages in multiple media formats. How does one search such an archive? How do experts interact with the content? How do users without any knowledge of language names, ISO codes, and language relationships find out information about the nearly 7,000 languages on the planet? These are some of the problems any project that claims to be "All Languages" must deal with. Our new archival structure is distributed and publicly interactive -- all languages and language relationships are available as open content in our Rosetta Base in Freebase, all of the archived materials are in our Rosetta Collection in the Internet Archive, and we are currently building a user-editable wiki interface on top of this (currently in alpha mode, so please ask me if you'd like a demo -- we also need a good name for it...Rosetta Panglossia?). A companion project to the digital archive is The Rosetta Disk -- a microscopic version of the collection, built out of materials that can last for millennia -- this is one of the showpiece artifacts to get people engaged in long-term thinking, along with the Foundation's 10,000 Year Clock of the Long Now.

Working Group 7: Collaboration Structure

Brian MacWhinney (chair)

wiki username: macw
website: talkbank.org
email: macw@cmu.edu
Brian MacWhinney, Professor of Psychology, Computational Linguistics, and Modern Languages at Carnegie Mellon University, has developed a model of first and second language acquisition and processing called the Competition Model. He has also developed the CHILDES Project (childes.psy.cmu.edu) for the computational study of child language transcript data and the TalkBank (talkbank.org) system for the study of conversational interactions.

Emily M. Bender, University of Washington

wiki username: EmilyMBender
website: http://faculty.washington.edu/ebender/
email: ebender@u.washington.edu
I am one of the co-organizers of Cyberling 2009. My interest in cyberinfrastructure for linguistics stems from my work on grammar engineering for linguistic hypothesis testing. I see this as one example of computational methods in support of linguistic analysis: using computers to systematically work with larger data sets and manage greater complexity than we could do without computational aids. I am also very interested in the issue of culture change within the field of linguistics, i.e., how to create a culture in which data sharing and the validation of hypotheses against large datasets are expected and rewarded.

Nicoletta Calzolari

wiki username:
website:
email:
[insert bio here]

Nancy Ide

wiki username:
website:
email:
[insert bio here]

David Robinson

wiki username: drobinsonlsa
website: http://www.lsadc.org
email: drobinson@lsadc.org
I am the Director of Membership and Meetings for the Linguistic Society of America. I will be attending Cyberling 2009 in order to assess how the LSA can best put the findings of this workshop at the disposal of LSA members as well as the profession at large, and what role the LSA can play in the development of a cyberinfrastructure for the profession.

Other Collaborators

Jeff Good

wiki username:jcgood
website:http://buffalo.edu/~jcgood/
email: jcgood@buffalo.edu
I am one of the co-organizers of the Cyberling workshop. (Unfortunately, however, I will not be able to attend most of it.) I am interested in how cyberlinguistic infrastructure can facilitate work in language description, typology, and comparative and historical linguistics.

Dan McCloy, University of Washington

wiki username: danmccloy
email: drmccloy@u.washington.edu
I am a graduate student in Linguistics at the University of Washington. My research is primarily in formal semantics. As one of the organizers of Cyberling 2009, I am the primary point-of-contact for most inquiries about workshop logistics.

Tandy Warnow

wiki username:
website:
email:
[insert bio here]

Towards a collaboration platform for linguistics

EmilyMBender — Sun, 13 Sep 2009 23:51:30 CDT

A call for tools.

Working group 3 was asked to discuss "tools for linguistics", with the possible secondary questions of identifying the properties of a "killer application" for linguistic research.

Our field (linguistics) has many tools already. We have annotation engines, analysis engines, and data-annotation standards. Sometimes, these tools are mutually compatible, but in general our tools exist for a sub-community within the field, and the tools work just well enough for the current research in that sub-community.

Currently open issues in technical tools for linguistics research seem to include:

How to collaborate (though Group 7 seems to be covering this)
Data provenance and proper citation of data-collections (Group 4 seems to be addressing some of this)
Data control (controlling access and controlling changes)

Our group's discussion made clear that there is at least one area of technical where linguistics -- as a field -- lacks a widely-used tool: in active collaboration on data collection, curation, and sharing.

A dream

Let's imagine what an active online collaboration tool might look like. We provide here a brief narrative suggesting what might be available for a linguistics community with such a tool:

Sissala studies (an imaginary tale)

August 1-20: Researcher S. makes recordings in field

September 3: 150 conversations uploaded to the shared research platform, tagged as well as S.'s notes allow. S. remains in Burkina Faso, but (because he logged in) his proprietorship of that data is clearly tagged. He sets the access permission on this data to be shared with certain other members of the community.

Sep 20: School begins in North America. An instructor's upper-level class in "Phonetics of Pitch" selects 5 of these conversations for annotation for F0 as part of the class projects. Some conversations are transcribed by more than one student. Each transcription is itself tagged with the identity of the student and his instructor, who herself is the one granting access to the specific conversations. In correcting the students' work, both the student's and the instructor's transcriptions and improvements are logged and made available back to S.

Four undergrad students studying IPA at another American university transcribe 15 sentences each, their teacher transcribes 30 and double-checks the students' transcriptions as well.

S., from Burkina Faso, is able to see these transcriptions and sends back some commentary, noting a phonological distinction that the IPA students may not have captured.

October 1: Parts-of-speech are attached by grad students in the Netherlands, who are doing typological studies of West African languages' word-order phenomena.

October 20: A syntactician in Australia takes note of new work on Sissala.

In less than an academic quarter, new data and rich linguistic annotations of that data are shared across four continents, among students, researchers, and faculty. The students are using the same tools as the faculty and the field researchers: classwork is a good preparation for fieldwork. The students' own practicum is made useful both to other students and to the original researcher, who may (or may not) have time to perform this transcription, but may be able to find the time to offer feedback as well.

In this (perhaps utopian) vision, research and education communicate using the same tools, and researchers across different sub-disciplines and different levels of expertise may also use these tools.

Working data sharing

What might such an active collaboration look like, using 2009 technology? Web programming and widely-supported data-distribution protocols make storing and distributing large amounts of data in a digitally-accessible way fairly easy. Creating, storing and managing data on server computers sounds complex (and is indeed) but these challenges are largely addressed by other communities with parallel needs, both commercial and non-commercial.

We envision an environment in which data-collection is (to draw an analogy) like constructing a post (or series of posts) for a weblog. The data (or post) may be stored on the server (which we semi-jocularly dub "cyberling.org" for this discussion) even before distribution (or "publication"); it may go through multiple drafts, and it is possible (as with blog posts, on some platforms) that only some readers may *ever* be able to read it; attribution and edit history is tracked, and distribution to other locations, authors, and/or readers is straightforward.

Contemporary web programming has ready-made solutions for many of these challenges. (This is not to say that the current solutions are ideal, only that they currently work well-enough.) Complete, well-tested strategies exist for backup (redundancy), storage, confirming and tracking identity, attribution, access control, and revision control.

Using these online environments also allows multiple researchers (possibly in different locations!) to extend and curate data collaboratively. These possibilities only richen as disciplines interact: for example, data collected for fieldwork in phonology and morphology may inform further research on sociolinguistics, syntax, or typology for researchers far away, if the disciplines can share datasets using an online collaboration. Collaboration in this manner can offer benefits to reputation (through citation of one's field notes and through visibility of one's own work) and new opportunities for researchers to share work in more repeatable, formally-visible ways.

Motivating existing researchers

While online data sharing strategies offer many additional exciting possibilities for collaboration, the "bootstrapping" problem lurks: existing researchers rarely have a strong interest in changing their current work patterns to join an as-yet-nonexistent collaboration network --- the benefits of collaboration will not appear unless many researchers are already involved; the first one on the dance floor will look goofy until the party really gets started.

Of course, additional objections may arise: there's not enough time or money; we've not done it before, why start now?, and the perennial concern: this is not our area of expertise. It is a truism of both open-source and for-profit software that no project will succeed unless it "scratches someone's itch" --- that it meets existing unmet needs of its users. In open-source software, the "itch" is often the desire of a capable programmer to improve her own tools; in for-profit software, the itch is the prospect that someone will pay for the product or the labor.

In this paper, we concern ourselves with discovering the "itch": as researchers, as collectors and curators of linguistic data, and as a community, we need individual (and collective) reasons to move towards this sort of sharing --- reasons to find the funding and the time to promote the kind of adjustment to our research methods that would allow this collaboration.

As researchers interested in sharing data, we want to encourage our users to join the collaboration, even before the benefits of the network effects appear. New users --- the "first ones on the dance floor" --- need good reasons to join before the network effects.

New users --- especially with the community in its infancy --- need good individual reasons to join. Thus, the question we address here is "why should I join a linguistics corpus-management service built on web tech?"

Case studies: Immediate Benefits

We believe that most linguistics researchers will benefit quite readily from this kind of tool built on web technology. In this section, we provide a sketch of several linguistics researchers who are fairly diverse in their subfields, interests, and experience, and we point out the benefits that each would incur from adapting his or her work to use these sorts of tools.

Sociophonetician

existing work: recorded conversations, annotated for a very narrow range of phonetic and sociological features. Current research tools: Praat, DAT tapes, and R for analysis.
Sharing data into the platforms suggested here gives her:
an offsite backup
shared access to the data with her research-assistants and annotators
control over who has access to these data
revision history (including opportunities to review and reconcile conflicting annotations)

Field lexicographer

existing work: documenting a language's lexicon in the field. Current research tools: Toolbox, and a paper notebook.
Sharing data into the platforms suggested here gives him:
backup
shared access to the under-development lexicon with his advisor (who is at home teaching)
having uploaded the lexicon in one format, downloading in a different one is easy
browsing the lexicon through the web
revision history

phonetics instructor

Teaching work: uses Praat. Students are required to learn enough computer tools to turn in IPA assignments in digital form. A perennial frustration: some submission formats corrupt the students' transcriptions.
By sharing some data to transcribe with the class, this instructor can ask his students to transcribe directly into the online tools described here. The students get:
practice with the tools that they will use as future professionals
work with real field data
attribution for their own work (their transcriptions are logged as their own)
The instructor gets:
extra annotation passes over the same data
clear indications of the students' transcription record (through the revision control and identity management)
standardized transcription responses

the doctoral student

Currently writing a syntax squib. Her research is currently looking at hundreds of Hungarian sentences, in search of left-displacement phenomena. The squib itself has only four (representative) examples.
If this student puts her examples into this sharing environment, she gets:
backup of her notes
organized documentation of the evidence behind that squib's point for later inclusion in her dissertation
reproducibility -- other researchers who might challenge the representativeness of her four examples can find other examples

the NLP evaluation guru

Currently running a Thai word-segmentation competition. Dataset involves thousands of sentences of Thai text, word-segmented by native Thai speakers. By using the platform provided, he can get:
data backup
easy indexing of multiple annotations of the same segments
easy revision tracking, as disagreements are resolved

Additional Benefits

A successful platform would probably provide access to several simple-to-use applications (such as tools to visualize the uploaded data). For example, visual and tabular reports of (word-)frequencies in your data, cross-tabularization, lemmatizers; visualization of syntactic annotation; simple annotation tools to add additional layers of annotation to a data set; and so on. Over time, community members can develop and share additional tools. Existing tools (e.g. ANVIL, Praat, Linguistic Search Engine; TigerSearch) could be integrated into the platform, combined with intuitive user interfaces, thereby providing additional motivation to linguists to join the community and upload their data.

In the long run, the platform we envision can also facilitate the development of annotation standards. Annotation standards that have been developed for one task can be become objects that can be shared with other users. Just like any other type of data, annotation schemes can have tags for authorship, editorship, and revision history. This way, not only primary linguistic data, but also secondary data (part-of-speech sets, syntactic annotation schemes, etc.) can be shared and improved by community members.

What to build first?

The system required to get all of the possible benefits of every one of these use cases will require substantial work. In this section, we propose a somewhat more limited scope as an initial goal.

We suggest that phonetic and phonological transcription are a particularly well-suited task for this sort of distributed data-sharing. Technology and standards are well-defined for sharing audio and transcriptions (essentially, text).

The issues of sharing audio and transcriptions among field researchers, their assistants, professional and student transcribers, and arbitrators are not simple, because they involve tracking meta-information, access control, and revision control. Software developers, for their own needs, have developed good tools for dealing with most of these issues.

Limiting the scope of this tool initially to audio files and multiple text transcriptions of those files would make working out these challenges somewhat simpler --- and thus more easily, in the future, extended to other forms of annotation (e.g., coding features that depend on other annotations, such as syntax above transcription).

Building the infrastructure for simple collaboration on transcription, however, would be a useful contribution to

Fieldworkers
Linguistics education - students and faculty
Sharing of transcriptions - between fieldworkers and their research colleagues
Collaboration among remote researchers (even when none of them are fieldworkers)

Conclusion

Too often, discussions about "tools" are really discussions not about tools, but about how to make our tools interoperate, how to make sure that doing work on tools or data gets proper attribution (or the related question: how it gets funding), or one of several other concerns that are -- at root -- questions outside of the tools themselves.

We have tools. The technology for sharing data, for managing access, revision history, redundancy, and privacy already exists, and is in use on the Internet every day. As the linguistics research community, we are not using the tools that exist for these tasks -- and some work must be done in order to use them to our (and their!) best ability. Nevertheless, the tools exist, and the next generation of linguists will thank us for already having them in place.

Group 1: Annotation Standards

mebeckman — Fri, 11 Sep 2009 12:26:35 CDT

Working group members: Mary Beckman (co-chair), Stuart Robinson (co-chair), Sarah Churng, Greville Corbett, Charles Fillmore, Richard Wright

Preamble

The Annotation Standards group was charged with identifying and documenting existing and needed standards for the annotation of linguistic data. Further, the group was asked to consider possible standards that may need to be developed in the future. Annotation standards support interoperability, aggregation of data, and (ideally) applications that help linguists address the research questions that they are interested in answering while creating consistently annotated data as a side-effect. Another consideration that the group took into account is that sometimes these goals may be in partial conflict with standards for the ethical treatment of human subjects.

This wiki page is the report ("white paper") from the group, who acknowledge the helpful comments of the other attendees at the Cyberling09 workshop, particularly Emily Bender, Nancy Ide, and Mark Liberman.

Table of contents

What is annotation and what is it good for?
What are annotation standards and what are they for?
What does it take to be a good annotation standard?
The state of the art (with some case studies)
Existing annotation standards and resources
References

1. What is annotation and what is it good for?

Annotation is the act of adding, to primary linguistic data, information representing analyses or models of aspects of the data. For example, if the primary linguistic data are an audio recording of a sequence of turns in a conversation between two speakers, then one type of annotation could be the marking of speaker-change points in the conversation within a layer of annotations related to the analysis of discourse structure. Another series of annotation layers could begin with an orthographic and/or a segmental transcription of the speech. Other annotation layers in this series might include a tokenization (segmentation) and glossing of the words or other similar units in the orthographic or segmental transcriptions of the recording. Other series of annotation layers on the morphosyntactic side of language could include a subsequent set of part-of-speech assignments to the words and/or a parsing of the syntactic structures of the sentences and other linguistic expressions in the recording. A parallel series of annotation layers on the phonetics/phonology side of language could include the tagging of linguistically significant events in the spectral patterns of the utterances (e.g., the release burst of each plosive and the transitions between different voice qualities), a parsing of prosodic structures that group segments into syllables and higher-order constituents, and the identification of salient points of coordination in the rhythms at different levels (e.g., marking of stressed or accented syllables).

1.1. What elements of an analysis can be annotated?

As the above example illustrates, elements of a linguistic analysis that can be annotated and for which annotation conventions can be codified separately are of at least three types: (1) tokenization/segmentation, (2) syntagmatic structure, (3) paradigmatic content of the events/tokens and structure. In the ontology of linguistic annotations, these aspects could be thought of as (1) the identification of instances or things, (2) the identification of relations among things, and (3) the identification of classes of things or relational functions. Bird and Liberman (2001) give an insightful discussion and a framework for formulating good ways of treating all three aspects. They propose to formalize this ontology in terms of the annotation graph -- a directed acyclic graph, in which each annotation token is (minimally) a triple consisting of two nodes that point to the positions in the string of labels on any annotation tier, and the label for the arc connecting these points, as in the two figures below.

Figure 1. Spectrogram (and Praat TextGrid merge of original label files) for the first three words in utterance train/dr1/fjsp0/sa1 from the TIMIT corpus (top) with a screen shot of the original phn and wrd label files (the first and third tiers of the Praat TextGrid) and the associated annotation graph snippet from Figure 2a in Bird and Liberman (2001).

Figure 2. Sample view for the first utterance in a Hayu narrative from the LACTITO archive (top) with a screen shot of the snippet of the annotation file and associated annotation graph from Figure 5 in Bird and Liberman (2001).

1.2. Examples

We expand on these three different aspects by illustrating each in reference to the type of example described above, where the primary linguistic data are an audio recording, and also in reference to cases where the primary linguistic data are instead a written text, which may or may not have begun as an orthographic transcription of an audio recording.

(1) For audio data, tokenization considerations can include not just the need to decide on the number of things that are instantiated at any given level of annotation, but also the need to agree upon where token boundaries (e.g., edges of segments or words) should be placed relative to the disparate spectral cues to the often asychronous and/or smoothly changing postures of different articulatory systems. For text data, tokenization considerations similarly can include the need to introduce word boundaries (e.g., as spaces in text) for languages with writing systems that do not use space for word separation, or, for English, the need to separate punctuation that shows significant syntagmatic boundaries -- by adding surrounding spaces -- from punctuation that is a part of a name or word (e.g., "that ' s that ! " as opposed to "etc.", "Mrs." "U.S.A.").

(2) For audio data, syntagmatic structure can include agreed-upon conventions for identifying time points on different annotation tiers that should be synchronized because they are the same event (e.g., the time stamp for the beginning edge of the first segment in a word is also the time stamp for the beginning edge of the word, and should change if/when the segment boundary is moved). It can also include the principles for differentiating different types of coordination across annotation tiers (e.g., when the time of a linguistically meaningful fundamental frequency maximum is identified relative to the time of a stop release). For text data, syntagmatic structure can include the bracketing of sequences of text that function as single constituents, or the indexing of anaphors to their antecedents, or the indexing of discontinuous collocates, as between the first and last words in "wreak this type of havoc".

(3) For audio data, the development of annotation conventions inevitably includes work to agree on the set of contrasting types to distinguish at several levels. For example, in tagging consonant and vowel segments, should the labels include only "broad" phoneme classes, or should major allophones be distinguished, or even finer phonetic detail marked? Whose intonational analysis should be adopted in tagging utterance melody? For text data, similarly, annotation conventions can involve work to codify the set of paradigmatic contrasts at many levels, from conventions regarding the number of different types of filled pauses and how to spell them in an orthographic transcription of recorded speech, to conventions for identifying cells in morphological paradigms, which might need to be differentiated in significance across languages. For example, systems for morphosyntactic glossing intended for cross-language comparison might need to recognize that for one language 'singular' is understood as in paradigmatic contrast to 'plural' while in another language it might be in contrast with 'dual' and 'plural'. This could conceivably be achieved by adding a "legend" to a given annotation layer, for a given language, linking each special annotation category to a description of the relevant contrast set, e.g., in a grammar.

1.3. The purpose of annotation and its relationship to the data

As the last example should make clear, conventions for each aspect of an annotation scheme cannot be established without thinking carefully about what the data are and what the annotations are for. In general, annotations are dedicated to specific purposes: it is hard to imagine a corpus development project that seeks to account for every detectable phenomenon in a language sample. Sociophoneticans who are interested in the increased use of creaky voice by young female speakers of American English would want to have an annotation tier where beginning and endpoints of creaky voice are marked; phonologists who are interested in comparing the probability of finding a particular segment sequence within a word to the probability of finding that sequence across a word boundary obviously do not need such a tier. Researchers interested in preferred or habitual locutions on the part of individual speakers obviously need to have speaker-ID information in the data they study; those interested in finding examples of syntactic phenomena obviously do not need such information. These different needs for more or fewer levels of annotation can be described as different points on a scale of granularity. But the different granularities can also involve the same level of annotation, but different degrees of specificity in the paradigmatic set.

The following is an example of levels of granularity in the syntactic description of English rate expressions such forty dollars an hour, forty miles an hour, forty miles a gallon, forty times a day, forty dollars an ounce, and the like. In dealing with such phrases, one purpose might be that of providing a preliminary mark-up for a parser, inasmuch as this pattern of two adjacent NPs is not a part of the ordinary grammar of the language. For this purpose, it would be enough to block off such phrases and mark them as NP: as such they then can fit into PPs (moving at forty miles an hour) and VPs (earns forty dollars an hour, gets forty miles a gallon), etc. A quite different purpose one might have for annotating such expressions is providing a mark-up that is usable for language understanding efforts: in such cases the type of unit (linear extent, money amount, time, weight, etc.) in each of the two parts of these expressions should be indicated (with information from a lexicon), allowing the automatic assignment of the phrases to such categories as Fuel-efficiency, Price-per-unit, Frequency, Speed and the like.

An obvious and important use of annotation is that of providing a layer of representation that is available for further analysis; in a sense, this amounts to regarding one person's annotation as another person's primary data. For example, phonological or orthographic transcription allows morphosyntactic analysis of a sample of speech more directly than the acoustic trace.

This property of layered annotation raises the issue, strongly associated with the late John Sinclair, that mistakes in one layer of annotation compound mistakes in higher layers. This point can be seen in the fact that descriptions of English syntax tend to accept the tokenization implied in the standard orthography, so that, for example, whose and another are treated as single units. Proposals about the clitic vs. suffix analysis of the possessive marker in English would be argued differently if whose were who's, making who the hell's fault is that? seem not so anomalous; and descriptions of the pattern that allows a mere twenty dollars, an extra five pages, an additional twenty dollars could be seen as incorporating another five pages (i.e., as an other five pages).

An analogous situation arises when phonological descriptions accept the tokenization or the set of paradigmatic categories implicit in the conventional segmental transcription for a language, even when describing speech produced by child speakers who have not yet acquired the phonological system of the language.

These pitfalls suggest that the sociology of developing annotation conventions might be an object of study in its own right. They also suggest that linguists should think flexibly about the types of things that should be considered annotations, so that conventions can be developed for how to link these things to the primary data. For example, it might be appropriate to think of responses from naive judges, elicited over the web using Mechanical Turk or the like, as a kind of annotation, in which case, it could be useful to develop standards for eliciting these judgments and tools for linking the responses back to the corpus of primary data that provided the stimuli. It might also be appropriate to think of skilled formant "correction" as a kind of annotation, in which case, there could be standards for "correcting" formants and associating the formant traces with the corpus, as in the development of the tools for the Origins of New Zealand English Project.

2. What are annotation standards and what are they for?

Some annotation is created to serve a single researcher's needs. If the annotation practices developed by this researcher, for a certain class of phenomena, are consistent, so much the better for this solo researcher. An issue of annotation standards arises when there is a need or opportunity for other researchers to work with the same data, or when researchers become interested in the same kinds of phenomena in other data samples, or in other languages, and want to be able to make generalizations.

An annotation standard, then, is a set of conventions that is associated with a commitment to adhere to the conventions by a community of users. A standard can evolve gradually in a community of researchers who are working on similar problems in some language domain, so that assumptions about the analytic space converge in some way that promotes the natural emergence of infrastructure for developing, transmitting, and codifying a standard. A standard can also arise from adopting a tool that brings with it assumptions about the data being analyzed that can be met by adhering to the standard.

No matter the path of convergence, however, annotation standards cannot be defined without reference to a shared set of assumptions and an associated community of analysts. As a corollary, a set of annotation conventions cannot be evaluated (or standardized) without at least an initial reference to a community of analysts and users.

Within such a community, annotation adds value by spreading the workload of providing agreed-upon analyses to a larger set of shared primary data than can be analyzed by a researcher working alone. Looking outward from the core community, annotation provides expert analyses for others who might not otherwise have access to the primary data.

This understanding of the relationship between the analyses of the data that are to be encoded in the annotations (the "model") and the primary data themselves leads to the following characterization of what annotation standards are for and how they can be evaluated.

2.1. Within the original community of developers and users ...

It is critically important to ground any annotation schema in terms of the particular model of the phenomena being annotated, and to develop it in relationship to the question being asked and the shared assumptions of the community about the phenomena being observed and modeled. Within this initial community, the annotations evolve as a set of "common law" rules about what the observed phenomena are. These rules will specify how the data should be segmented into tokens, how the tokens will be labeled in terms of an agreed upon inventory of contrasting types and relationships, and how relationships among tokens will be parsed and labeled.
A set of annotation conventions (rules), therefore, can only be evaluated first in relationship to the initial user community and their questions. While there might be domain-specific evaluation criteria, defined relative to independent observational tools, one critically important evaluation criterion that is common to all domains is the reliability of the annotations. Is the annotation consistent within and between annotators? If an annotator observes the same data twice in different independent annotation sessions, are the analyses (the annotations) the same? Similarly, if two different annotators observe the same data independently, do they arrive at the same analyses (tokenization and labelings)?
To achieve consistency typically requires a long iterative process of "common law" development, during which two or more users annotate some set of data separately, then convene to discuss and adjudicate the disagreements, formulate new principles to cover the cases discussed, and then start a new round of independent annotation, comparison, discussion. The initial users will need to agree on the degree of consistency that is needed to accomplish their goals. An ancillary set of "laws" will need to be developed to reliably differentiate between disagreements that arise from intrinsic ambiguity and disagreements that arise because the conventions and annotation tools are not yet at the point of required coverage/stability/useability.
It can be useful in the process of developing, evaluating, and using an annotation standard to work on the different aspects separately. For example, if at some stage of development (or in some subcommunity of users), the tokenization is more reliable than the identification of relationships, the annotations might be adequate for some subset of the initial purposes, but not others. It is then important to develop conventions for tagging corpora or parts of corpora for relevant facts such as which version of the annotation scheme was used or the level of experience and/or training of the annotator(s).

2.2. When extending to a new community of users ...

Annotations developed within a particular community of initial developers and users might be extended to another community of users who are addressing different questions and may have different model assumptions. The goodness of a standard then becomes a product not just of the initial developers/users, but also of the flexibility/ingenuity of the later users of annotated data.
The needs of various communities are in some cases overlapping (both phoneticians and sociolinguists may seek standards for phonetic annotation) and in other cases conflicting (a fieldworker may want language-specific idiosyncratic part-of-speech categories in interlinear glossing whereas a typologist may want agreed-upon cross-linguistically motivated categories). To get a sense of the potential disparaties among different communities of users, we listed the first sets that came to mind:

sociolinguists
computational linguists and NLP practitioners
language acquisition specialists
psycholinguists and laboratory phonologists
specialists in speech and language disorder
fieldworkers and language typologists
stylometrists, disputed-author researchers, etc.
educators evaluating text complexity, comprehensibility

Even within a single later-adopting community, however, the questions and needs may differ in relationship to different types of primary data. Here we can differentiate at least among (1) full video recordings, (2) audio-only recordings, and (3) spoken utterances that were recorded only as text in the first level of "annotation" of the fieldworkers' written transcription. The initial tokenization/labelling of each of these primary data types may be an orthographic transcription, and in the case of type (3), the initial tokenization/labelling then becomes the only record. For some communities of users, the models and questions related to such transcriptions might differ dramatically from the models/questions that can be applied to data that are (4) originally written texts.
In adopting (and adapting) a set of annotation conventions to a new set of questions and applications, then, it is again useful to ask: What aspects of the annotations can we usefully tease apart and evaluate/adopt/develop separately?

3. What does it take to be a good annotation standard?

3.1 Best practices (themes)

The associated properties that define a good annotation standard can be grouped into a few overarching themes and associated questions about the annotation conventions:

Consistency/Reliability

What is the history of the annotation conventions? Did they evolve in careful, iterative rounds of (1) discussion of the goals of the annotation set, (2) independent annotation of a suitably diverse corpus of primary data by a large number of annotators, (3) calculation of inter-annotator agreement, and (4) discussion of points of agreement and disagreement and incremental revision?
Are there standards / mechanisms for continued calibration of consistency within and between annotators?
What are the published intra-annotator and inter-annotator consistency rates?
Are the conventions designed to allow transparent, easy, reliable "back-tracking" to the primary data, via time stamps or via sequence position nodes within an annotation stream that has a reasonably fine-grained tokenization?

Useability

Is there good (accessible and extensible) documentation?
Is there a suitably diverse and continuous community for teaching (and testing the ability of) new annotators / users?
Are there good tools for annotating and using the annotations, and good community mechanisms for building / extending / sharing tools?
Is there a reliable connection between the annotations and the primary data that allow the user to track back to the data to check a suitable subset of the annotations?
Is the design of the annotation schema such that annotations can be used as reliable tags back into the primary data, for easy queries using standard query tools?

Resilience

How does the standard deal with inter-annotater disagreement? Is information about disagreements preserved so that they can be analyzed in the course of developing the next version, to determine whether there are common cases of inherent ambiguity that need to be marked, or new cases that the conventions do not yet cover?
Are there principled mechanisms for marking degree of uncertainty about difficult or ambiguous cases? (See the CHAT manual for a thoughtful discussion of this question.)
Are there graceful ways of choosing to provide more or less specific degrees of analysis?
Are there good mechanisms for providing and getting the most out of partial annotations?
Are there robust ways of extending partial annotations to more of a corpus and of verifying and modifying the annotations of a corpus?
Relatedly, are there good mechanisms for keeping track of which parts of a corpus are in what state of annotation and verification / modification?

Accountability/Responsibility

Again, are there robust mechanisms for maintaining transparent links back to the primary data? and are these mechanisms ethical? Do they insure the explicitly or implicitly agreed-upon degree of confidentiality of the person or people who produced the primary data (or who produced some subset of the annotations by providing naive judgments)? The issue of confidentiality is especially vexing when the primary data are video recordings. (See sections 4.3 and 4.4.)
Do the standards encourage (or even allow) later "consumers" to credit the annotations in publication?
Are the annotators (or the annotator level) for different parts of a corpus or different aspects of the annotation identified in a way that allows later users to partition the annotations -- e.g., into annotations by native speakers versus non-native speakers?

Interoperability

Can the annotation be validated and used in different tools or computational models?
Is the logical structure of all three aspects of the annotation conventions transparent, and transparently related to the documented descriptions of the annotated phenomena?
Also, is it possible to translate to and from some other annotation conventions that have been used for this set of phenomena, in a way that makes it possible to share data across different analytic frameworks?
Are the formats for encoding the different aspects of the annotation conducive to using the annotations for purposes different from the originally intended ones?
Are the definitions of the annotation elements freely available and stored in an open format?
Are any requisite tools for annotating or using the annotations free open source?

Extensibility/Adaptability

Can the annotation schema be extended to annotating utterances in other styles from the utterance sets for which it was developed? Can the annotation conventions be used for utterances produced by other speaker types? Can they be extended to (or readily adapted for) annotating data from other dialects, other languages, ....?
Is there a solid and suitably diverse core of users (and "maintainers") to allow the standard to evolve and change in response to user feedback and/or to new needs?
Is there a sensible consensus or mechanism for deciding when to "publish" a new version?
Are there good standards and mechanisms for versioning? For example, is there a robust way to permanently associate meta-data about which version of the conventions was used in annotating (different parts of) any corpus? Are there tools for keeping track of who the taggers were at different levels / times, and are there tools (or at least a "crib") for how to "translate" across corpora and/or across levels of annotation as the standard evolves and expands?

3.2. Best practices ("tangibles")

The hallmarks of an emerging annotation standard therefore begin with these two important social characteristics:

community

There is a sustainably large and diverse community of core users/maintainers.

history

There is a history of effective dissemination of the conventions and recruitment of new core users.

Other more tangible accoutrements of annotations standards that have exemplified the best practices identified above include:

documentation

The conventions are adequately and fully documented, in a "reference manual" that can be consulted easily by experienced users.

training manual

Ideally, there is also a separate, well-tested training manual (or a standard syllabus for training courses) that leads new users through a graduated sequence of more and more difficult examples culled from data that were annotated in developing the documentation and/or the reliability metrics.

inter-annotator reliability metrics

There are published records of inter-annotator consistency tests. Ideally, these tests differentiate between disagreements that stem from intrinsic ambiguity and disagreements that have other, remediable sources such as inadequacies in coverage, deficiencies in the documentation, or the like.

computational tools

Members of the community have invested in developing computational tools that increase the reliability of the annotations.

conventions for metadata

The community has developed mechanisms for protecting confidentiality of the producers of the data, crediting of the provenance of the annotations, and so on.

conventions for responsible maintenance

A set of conventions, or a more elaborate institutional framework, has also emerged, for responsible maintenance of the conventions, for continued elaboration of the documentation, and for updating of the training manual (or re-accreditation of the training courses).

4. The state of the art (with some case studies)

In this section, we illustrate the considerations outlined in Section 3 by briefly reviewing the development and current state of annotation standards in four very broad areas. These reviews highlight two factors that promote or hinder the development of reliable and resilient standards.

The first is the degree to which the "semantics" of the target phenomena are naturally constrained. At one extreme is the case of phonological annotation of consonants and vowels of spoken languages. Here, the aerodynamics of the vocal-auditory channel tightly constrains the tokenization and types of relationships at the lowest level of the prosodic hierarchy in an extremely robust way. An example at the other extreme is the analysis of grammatical constructions, where it is difficult to even imagine what boundaries could be imposed by nature on what a language-specific morphosyntactic construction can mean.

The second factor is the age of the language type and/or of the systematic linguistic investigation of the phenomena across languages. Spoken languages may have existed as long as there have been modern homo sapiens, and the "annotation" of consonants and vowels goes back to the first alphabetic writing systems. By contrast, the systematic study of signed languages has a much shorter history.

4.1. Phonology of spoken languages

4.1.1. Annotation systems for vowels and consonants

As noted above, tokenization and other aspects of the analysis of categories at the lowest level of the prosodic hierarchy for spoken languages is tightly constrained by the (psycho)physics of the human articulatory and auditory systems. As a result, it has been relatively easy to develop conventions for annotating utterances of spoken languages at this level of this part of the grammar, using an alphabetic analysis, and the International Phonetic Alphabet is a premier example of a well-developed annotation standard. It is maintained and updated by the International Phonetic Association, which was established in 1886 and today is associated with the International Congress of Phonetic Sciences, a meeting held every four years which attracts several thousand attendees.

The International Phonetic Alphabet has a handbook, which documents the annotation conventions and specifies a well-codified format for presenting a catalog of the consonant and vowel inventories of a spoken language variety using the IPA. (There is a long-standing section of the Journal of the International Phonetic Association devoted to publishing such language-specific schema.) The most recent version of the handbook was published in 1999, after a conference convened in 1989 to review the coverage of the categories in the IPA consonant and vowel charts and the lists of other symbols for categories that do not fit neatly into the tokenization and paradigmatic features that are encoded in the consonant and vowel charts. In between the conference and the publication of the revised Handbook, the Journal of the International Phonetic Association published a report on deliberations of the conference as a whole (JIPA, 19: 67-80) as well as reports from working subgroups charged with deliberating on various more focused issues such as "Computer Coding of IPA symbols" (JIPA, 19: 81-82) and "the best means of transcription of disordered speech" (JIPA, 24:95-98) and correspondence from members of the association commenting on the proposed revisions and deliberations at the conference (e.g., JIPA, 20: 22-32).

While there is no official training manual, there is a long history of teaching the annotation conventions (and the phonological analyses on which they are based), which predates the founding of the International Phonetic Association. For example, Henry Sweet's A handbook of phonetics, published in 1877, includes vowel and consonant tables that are organized in terms of the same dimensions of analysis as the modern IPA chart -- i.e., openness, place, rounding for vowels and place, manner, laryngeal properties for consonants. The same basic approach is also taken in most subsequent textbooks, including Peter Ladefoged's well-known A course in phonetics, which is still used in training students in the annotation of vowels and consonants in many departments of phonetics, linguistics, logopedics, and speech & hearing science.

There is also a history of research on inter-annotator consistency rates for segmental transcription, and on the factors that affect transcription consistency. In general, it is easier to be consistent the closer the annotation is to a "broad" phonemic transcription. For example, Eisen (1993) reports complete agreement among three transcribers of only 50% for a "narrow" transcription, even when distinguishing among only ten "major class" categories such as "voiced plosive". When the same transcribers were asked instead to note only segments that deviated from an automatically inserted broad "dictionary" form transcription, consistency improved to 85%.

4.1.2. Where the analysis breaks down

The different levels of reliability for "broad" versus "narrow" transcription hint at some of the things that affect inter-annotator reliability. Reliability is highest when the primary data are clean recordings of careful fluent utterances produced by adult native speakers of a dialect for which there is a consensus phonemic analysis that can be the basis for the tokenization and consonant and vowel label set, and when the goal of the annotation is to produce a "broad" phonemic transcription as the basis for morphological analysis or the like. Reliability is lower when recordings are noisy, when the primary data are casual or dysfluent utterances, when the dialect of the speaker(s) is an understudied variety that differs from the dialect on which the IPA description is based, or when the goal is to produce a "narrow" transcription as the basis for sociophonetic analysis of variation in the speech community or the like. In the latter cases, reliability is improved if annotators are well trained in phonetics (not just in "classical phonemics") and have recourse to tools such as the interactive spectrographic display window in the Praat signal analysis tool. However, no amount of phonetic training will resolve the inherent unreliability of shoehorning "sub-phonemic" paradigmatic variation and "suprasegmental" parsing differences into a phonemic segmental model.

Alphabetic annotation of pre-school children's speech poses special challenges, then, because it assumes (paradigmatic and syntagmatic) phonological structures that may not be in place until the child is much older. Very cohesive research groups, such as the Stanford Child Phonology project that collected and annotated a cross-lingusitic longitudinal corpus between 1967 and 1992, can achieve published inter-annotator consistency rates as high as 90-95% agreement. However, this agreement is typically achieved by regular (at least weekly) meetings among the primary transcribers, during which inconsistent tags are "corrected" to a consensus category (or to an "expert" tie-breaker category when consensus is impossible). Pye, Wilcox, and Siren (1988) suggest that this practice hides the true nature of the difficulty, since points of low inter-transcriber reliability can indicate places where the standard phonemic analysis of the target language is particularly inappropriate for the child's developing phonological system. Hewlett and Waters (2004) make a similar point, suggesting that the problem of obscuring "sub-phonemic" variability is compounded in most large-scale cross-sectional norming studies, in which fairly "broad" transcription is used in on-the-fly observations without a permanent audio recording, in order to be able to collect data from a large number of children. Edwards and Beckman (2008) suggest that even in cross-sectional studies where "narrow" transcription of recordings is done, transcription should be supplemented by experiments eliciting (potentially continuous) perceptual responses from phonetically untrained native speaker/listeners. All of these researchers remind us that tokenization and paradigmatic differentiation at the level of vowels and consonants is a product of the interaction of the natural constraints from the aerodynamics with the exigencies of lexical contrast in dense neighborhoods. The phonemic analysis that is the basis for the IPA conventions is less compelling for a speaker whose lexicon is still too small to have very dense phonological neighborhoods.

Prosodic phenomena such as stress and syllable structure, and intonational phenomena such as the melodies that group syllables together into phrases and the like, pose a related challenge for phonological annotation. Because there is no comparably compelling natural basis for tokenization of melodic events, spoken languages are much more diverse in the ways in which utterances are structured above the leaf nodes of the prosodic hierarchy. The Working Group on Labeling of Suprasegmentals at the 1989 conference that led to the current IPA handbook recognized this by deciding to recommend no standard annotation conventions for intonation (Bruce, 1989). A basic principle of the ToBI annotation framework also says that "phonetic transcription" of prosody and intonation is impossible. See Beckman, Hirschberg, and Shattuck-Hufnagel (2005) for further explication of this point and the implications for the development of annotation conventions for prosody and intonation. See also Pitrelli et al. (1994) for the remarkably good inter-annotator reliability rates that nonetheless can be achieved when conventions are specific to a particular dialect.

4.2. Morphosyntax and semantics

In the preceding section we described how phonological and phonetic annotation is easier for broad class distinctions at the leaf nodes of the prosodic hierarchy, where tokenization is naturally constrained by the psychophysics of speech production. There may be an analogous difference in the degree of difficulty of annotation for morphosyntactic structures and semantics of spoken languages. Specifically, it again seems easier to agree on tokenization of elements that are closer to the bottom of the constituent hierarchy, where the psychophysics of the vocal-auditory channel interact with more general cognitive considerations of attention and memory to promote a "temporal" or sequential (as opposed to a "spatial" or simultaneous) decomposition of the form/meaning mapping. We illustrate by describing standards for annotating aspects of morphology, syntax, and semantics that have emerged in three different communities, before touching on some of the challenges in these areas of the grammar.

4.2.1. The Penn Treebank

The Penn Treebank is a collection of English texts that have been grammatically annotated for part of speech, grammatical function (predicate argument relations), and constituency (Marcus et al., 1993; Marcus et al. 1994; Taylor et al., 2001). For each sentence there is an accompanying syntactic analysis (a tree), hence the term treebank (a bank of syntactic trees). Although the Penn Treebank contains material taken from multiple sources (the Wall Street Journal, the Brown Corpus, the Switchboard corpus, and ATIS), its collection of annotated Wall Street Journal newspaper articles is so well known that many researchers think of it the Penn Treebank as the Wall Street Journal corpus. It is available under a commerical license from and distributed by the Language Data Consortium (http://www.cis.upenn.edu/~treebank/). The corpus is distributed as a collection of texts with accompanying stand-off annotation. The Wall Street Journal section of the corpus, for example, consists of 2499 articles (totalling approximately a million words) published in the Wall Street Journal during a three year period. For each article, there are a number of plain text files that contain various types of annotation as well as a "master" annotation file that integrates all of the annotation.

The Penn Treebank is an important point of reference in grammatical annotation given its success. Not only has it become an important resource in computational linguistics (like WordNet or CELEX), as any search of the literature will reveal, but it has also inspired a large number of similar projects for other languages--e.g., Chinese (Xue et al., 2002), Czech (Hajicova, 1998), Spanish (Navarro et al., 2003), and German (Brants et al., 2002).

The Penn Treebank Project has a number of strengths that help explain its popularity. Chief among these, of course, is that by providing a non-trivial amount of annotated newspaper text it managed to scratch an itch felt by the community of researchers interested in computational linguistics, natural language processing, and related fields. But in addition the Penn Treebank Project provided good documentation and designed the corpus in such a way that it could be easily used: consisting only of plain text files (easliy processed), providing good documentation, versioning the corpus, etc.

Despite its success, the Penn Treebank has its weaknesses. One of these is the absence of a standard toolkit or user application for its viewing and/or manipulation. This is unfortunate since such a toolkit could have improved the pace of development and helped improve annotation quality (by eliminating errors that could be detected through automated validation). The under-availability of tools also impedes its adoption by those lacking the resources to develop their own tool. (The creation of various open source toolkits has ameliorated this problem to some extent, but it is a comparatively recent development compared to the age of the Penn Treebank Project.) Another potential weakness is the grammatical model used to describe grammatical functions, which adheres to an old-fashioned Government and Binding analysis that posits, among other things, traces for movement. However, even though the annotation is couched in a multistratal theory of grammar, this has not hindered its use in monostratal theories of grammar, such as LFG (Frank, 2000).

In fact, it is unrealistic to expect a great deal of standardization of the content of annotation for grammatical information given the highly contentious nature of grammatical theory itself. The difficulties inherent in this problem can be seen in attempts to develop treebanks in languages with more flexible word order and discontinuous constituency, such as German. Although it is possible to treat annotate German using a grammatical theory that posits traces, it leads to inelegance and German treebanks have as a result departed from its grammatical model. (The NeGra and TiGer annotation schemes use graphs with crossing edges rather than simple context-free trees.)

4.2.2. Leipzig Glossing Rules

The Leipzig Glossing Rules, which build on Lehmann (1983) are a de facto standard for morphosyntactic glossing. They represent a light-touch codification of previous best practice. They allow for analyses of varying levels of granularity, subject only to the requirement that the segmentation in the glossing line must match that in the source language line. There is a standard set of abbreviations, which could usefully be extended. Documentation consists of a short document, with examples, freely available on-line.

In their current version, the Leipzig Glossing Rules provide the means and the options for linguists of different persuasions to give adequate morphosyntactic glosses. A next step would be to suggest how users should characterize their own usage in a particular publication. At the most obvious level, all additional abbreviations should be specified. Then, we should note that the abbreviations are mainly for feature values (for ‘singular’, ‘feminine’ and so on); it would be good practice to specify which feature each is a value of. Normally this is obvious, occasionally it can cause confusion. Finally, since the rules can be used for different purposes, it is helpful if users clarify and spell out their assumptions. For most purposes, when glossing yesterday we bid three hundred pounds for that horse, we would gloss bid as past tense. The information is derived not from the form itself but from the time adverbial, since the form bid could be present or even imperative. For writing about tense, word order, argument structure, and so on, this solution is fine. If writing about syncretism, however, this theoretical ambiguity would matter, and would need to be indicated appropriately. More generally, the annotator frequently has to be selective in the level of detail included in the morphosyntactic glossing. This means that we cannot expect that different linguists would provide identical annotations. But we should aim for a greater level of consistency than we often find. The rules offer the alternatives, but good practice requires us to choose consciously, to specify the choices made and to apply them consistently.

In terms of tools, it would be useful to have a tool that would check annotations for internal consistency and for any unintentional departures from the conventions.

4.2.3. FrameNet Annotation Criteria

The Berkeley FrameNet project is using annotation of written corpora to build an on-line lexical resource for English, based on frame semantics and supported by corpus evidence.

Certain annotation practices followed in FrameNet can be seen as arbitrary but motivated. One of the central notions of FrameNet is that of Valence: general descriptions of the combinatory possibilities of individual lexical heads (verbs, nouns, adjectives, and some prepositions), expressed in both syntactic and semantic-role terms. Valence descriptions are derived automatically from a body of annotated sentences, so it is obviously necessary to agree on how the annotations are structured. The need to pair syntactic arguments with semantic roles motivates our decision to include the "markers" of a phrase with the constituent. For example, if we wish to identify the speaker and the content of an announcement from the phrase the announcement by the governor of her decision to resign, those two elements are blocked off as [by the governor] and [of her decision to resign]: some projects would ignore the prepositions and select only the NPs in those expressions. Similarly, in the governor's announcement that she intended to resign the labels would be assigned to [the governor's] and [that she intended to resign], including both the genitive suffix and the that-clause. And similarly, then, in [the governor] announced [to the world] [that she intended to resign]. A differently motivated project might leave out the structure-markers in order to represent the "content" more faithfully; FrameNet includes these in order to match semantic and syntactic segmentation, allowing users to recognize the structural elements.

It will be seen that the annotations themselves do not distinguish markers that are determined by the grammar (the possessive ending in the governor's announcement), from those that are determined by the meaning of the PP as a whole (the preposition in under the table), or by the governing lexical head (the preposition on in we can depend on Harry. Such information is recoverable from the grammar and the lexicon, but it is not part of the annotation.

The standardization of this FrameNet practice has various consequences. All annotators on the Berkeley project agree to use it in their work, and various FrameNet or FrameNet-like projects in other languages have agreed to follow the same, or analogous conventions. These are Spanish FrameNet, Japanese FrameNet and the SALSA project in Germany. Furthermore, Professor Hiroaki Sato of Senshu University in Tokyo manages a browser of FrameNet data ("FrameSQL" http://sato.fm.senshu-u.ac.jp/fn2_13/notes/index.html) and he is developing a way of pairing valence patterns across the various languages that have FrameNet databases; the comparisons work best if all users treat function markers in the same way.

4.2.4. The difficulty of developing annotation systems for grammatical constructions

There are well-known tree-banks that offer syntactic parses of all of the sentences in a sample, meeting certain levels of adequacy, such as the Penn Treebank. There are various levels of part-of-speech tagging for large corpora, such as the British National Corpus, that are for the most part successful. But a complete record of the special grammatical constructions in a text does not seem feasible.

For research purposes it should be possible to tag (say) all comparative sentences in a text, identifying the scales and the phrases that directly or indirectly indicate the entities being compared. It should be in principle possible to identify all idioms or tight collocations in a text, however long this might take. It should be possible to notice constructions with certain peculiarities for the sake of assembling examples for further study, such as, for English, the pattern that has a degree-modified adjective followed by an indefinite NP marked by the preposition of. (Do you need {[this big] [of a box]}?) But the expressions that represent individual constructions are frequently tightly intertwined, and the effort to work out the nature of such integration on a large scale is not likely to be possible. A sentence like He's in no bigger of a hurry than you are exhibits a comparative structure (bigger ... than you are), a collocational idiom (in ... a hurry, one of the few uses of hurry as a noun), the puzzling structure with the of-phrase (bigger of a hurry), a special minimizing use of the word no with a compared adjective (consider the difference between he's not smarter than your mother and he's no smarter than your mother), and the particular form of the than-clause (than you are vs. than you, than expected, than ever, etc.). Representing the working of all of these constructions and their articulation is not to be expected.

Research that collects and explores examples of grammatical constructions, idioms, collocations, and multiword expressions in general, and illustrates their properties one at a time, has got to be an essential task for linguistics and computational linguistics, both for grammar writing and as a way of producing learning corpora for machine-learning techniques to improve syntactic parsers. But since many of the most important constructions cannot easily be associated with individual words in a sentence, or with specific nodes in a parse tree, there is little likelihood of acquiring large-scale accurate annotatons of grammatical constructions, beyond familiar parsing and chunking of nonproblematic sentences, any time soon. The problem is further compounded by the fact that cross-framework agreement on syntactic phenomena in general is not easy to achieve: dependency-based and constituency-based treatments are not always interconvertible; theoreticians who seek to minimize redundancy in their analyses would not see the same number of construction types in a given text as the grammarian who wishes to work with structures of finer granularity.

The proposal in this section favors rich analysis of small texts, together with extensive sampling of given constructional phenomena one at a time, or in small families of constructions. Such a combined approach should eventually lead to understanding the importance of non-core constructions and multiword expressions, classifying their variety, estimating how many of them there are, determining their relevance in profiling different genres, estimating their "density" in different kinds of texts, exploring the manner in which they are learned, and evaluating their contribution to measures of language complexity.

A sample of constructional annotations prepared within the Framenet project can be seen on http://www.icsi.berkeley.edu/~hsato/cxn00/21colorTag/index.html.

4.3. Annotation of gesture

The two factors relevant for annotation developments, as discussed above in the beginning paragraphs of Section 4, are especially salient for the progression of standards in gesture annotation. We discuss gesture here as inclusive of gestures in signed languages as well as discourse-related gestures of the spontaneous type which accompanies natural spoken language narratives.

First, in contrast to the tightly constrained audio-articulatory modality of spoken language systems, which are articulated using the vocal tract, gesture systems involve the visual-gestural modality. Traditionally, gestures are understood to be articulated using the hands in movements. In the case of sign languages, however, the category of gestures has recently been expanded to include certain movements involving the head, face, and shoulders. (See Neidle et al. 2000 for some discussion of nonmanual gestures in ASL, and Boyes-Braem, 2001 for the descriptions of "mouth gestures" in multiple European sign languaes.) Moreover, the visual space of gestures in sign languages exists as a complex continuum which involves the signed phonemic structures in the lowest level of the prosodic hierarchy, larger signed morphemes which engage in spatially bound agreement relationships with other signs, and "nonmanual" gestures of higher-order prosodic structures. So while the conventions for annotating gestures in the traditional sense may be developed with relatively straightforwardness using video analysis, developing annotation standards for gestures which handle the complexity of these relationships must also rely on multiple and dynamic layers of annotation of the types illustrated in the figures and examples in Section 1.

The Language Archiving Techonology tool, ELAN, is a professional tool for the complex annotation of video and audio sources. "Tiers" are implemented for simultaneously displaying and annotating parallel levels of analysis. These can be nested for dependencies between, say, an independent parent annotation of morpheme-by-morpheme transcription, and referring tiers for the varying gestural articulators (e.g. hand vs. mouth). A full manual for ELAN is available online.

Second, the systematic investigation gestures as a linguistic phenomena is a relatively new pursuit. This is true for both the annotation of gestures in spoken language narrative studies (such as with the McNeill Lab project) as well as the annotation of sign languages (see Neidle et al.'s Sign Stream Project, and the Berkeley Transcription System manual). The next section presents some considerations for developing unified conventions in the annotation of sign languages.

4.4. A unified annotation standard for signed languages

Finally, we address the need for a unified annotation of sign languages, as identified in this workshop group and codified within the desiderata for qualities of annotation standards in general. First, however, some prior discussion concerning the dissemination of tools and standards among the the communities of practice (sign language linguists) is necessary. For although several tools currently exist for the scientific annotation of video data (see above for gesture annotation using Anvil, ELAN, and Cross-Modal Analysis of Signal and Sense, for example), and while the target users are a close-grained community, widespread standard for sign language transcription and annotation is lacking.

4.4.1. What sign language annotation is, and what it is not

We begin by clarifying the purpose of a unified standard of sign languages. We emphasize that we do not aim to advocate a writing system of signs, nor do we intend for annotation to replace the primary linguistic video data with a derived set of data. Rather, annotation of sign languages should complement the primary data record as a way of tagging and searching the data. And the goals of a unified sign language annotation standard are to provide a shared platform of convention for collaborating across the various linguistic domains.

4.4.2. What a unified standard provides

For all linguists, annotation is paramount, and standards promote convergence. For sign language linguists, the annotation of primary video data poses several challenges. Few standards exist, for example, when it comes to annotating sign languages for fundamental linguistic phenomena such as pronominalization or indexicalization within the interlinear gloss. A more complex issue is the matter of transcribing certain non-manual features that are coarticulated with the manual signs--as functional labels (neg), abbreviations of the action (head shake), or even further break-down of the correlates involved. On the practical side, high-quality video data can require large loads of memory, and utilizing tools for analyzing video requires higher processor speeds and memory load.

These issues stand to hinder the standardization of data annotation. Through further practice and dissemination, however, advances in sign language annotation standards provide the potential for consistency, conversation, and conventionalized practices among a growing community. The ideal situation (projected solution) is one where sign language linguists, whether collaborating in an international workshop setting or via remote communications, have access to one mutually accessible standard that is extensible for all sign languages, interoperable across varying domains and models of interest, granular across levels of linguistic analysis, and practical for continuous usability.

5. Existing annotation standards and resources

This section lists the various annotation conventions and other resources for developing and discussing annotation standards that were suggested by participants of Cyberling09.

5.1. Phonetics and phonology

Phone segment tagging symbols, including both:

IPA and its various ASCII-fications, such as SAMPA, WorldBet
and language-specific phoneme-segment encodings such as ArpaBet for American English and the CSJ encoding for Japanese

The various ToBI conventions and similar conventions in other frameworks such as the ToDI conventions
The PhonBank conventions and tools

5.2. Morphosyntax

Leipzig Glossing rules
ISO Morphosyntactic Annotation Format (MAF)
ISO Lexical Markup Framework (LMF): Homepage (with Publications and Tools)
Typecraft: a labeling system which, for any verb construction of a given language, provides a template for that construction type displaying its argument structure, in a fashion as transparent as possible. The template is constructed from a universally established inventory of labeling primitives.
http://www.typecraft.org/tc2wiki/Verbconstructions_cross-linguistically_-_Introduction
tags for short-unit word (SUW) and long-unit word (LUW) in the CSJ

5.3. Syntax and semantics

FrameNet annotation manual
The tokenization guidelines, part-of-speech tags, and bracketing conventions for the Penn TreeBank project
ISO Syntactic Annotation Format (SynAF)
ISO Semantic Annotation Format - Time and Events (SemAF-TIME) (formerly TimeML)

5.4. Pragmatics and discourse structure

CHAT conventions for segmenting turns and identifying the participant and the setting
DAMSL
and other schemes documented and discussed at the 1998 DRI meeting such as:

intentional structure annotation (see Nakatani, Grosz, Ahn, and Hirschberg, 1995)

5.5. Gesture

David MacNeil's Gesture annotation
BTS sign transcription system
NEUROGES-ELAN system: workshop series homepage
Michael Kipp's Anvil
Carol Neidle's SignStream
Vislab's (Francis Quek's) Cross-Modal Analysis of Signal and Sense for multi-modal human discourse

5.6. Other resources

The EMU Speech Database System (see Cassidy and Harrington, 2001): sourceforge page
ISO TC37 SC4 - Language Resource Management : Homepage

ISO Data Category Registry for linguistic concepts

GOLD (General Ontology for Linguistic Description)

various (lineages of) POS tagging systems such as:

the ones assumed in the taggers linked into the Stanford NLP resources page

text transcription symbols promoted on the LDC Corpus Cookbook page for transcription/symbols
International Standards for Language Engineering (ISLE)

ISO Linguistic Annotation Framework (LAF) Overview and Draft Standard (under revision)
XML Serialization for LAF: ISO Graph Annotation Format (GrAF)
SoundIndex and related search and view tools developed in the LACITO Linguistic Data Archiving Project (see Jakobson, Michailovsky, and Lowe, 2001).

XML Corpus Encoding Standard (XCES)

SIGAnn : ACL Special Interest Group on Annotations

NXT System: Switchboard in NXT NITE XML Toolkit Video lecture by Jean Carletta: the NITE XML Toolkit Meets the ICSI Meeting Corpus

tool and framework for building

E-MELD (Electronic Metastructure for Endangered Language Data): Homepage

All of the annotation systems listed on the COCOSDA Corpus Annotation Tools page
The Automatic Mapping Among Lexico-Grammatical Annotation Models project resources

6. References

Beckman, Mary E., Julia Hirschberg, and Stefanie Shattuck-Hufnagel. 2005. The original ToBI system and the evolution of the ToBI framework. In Sun-Ah Jun, ed. Prosodic Typology: The Phonology of Intonation and Phrasing, pp. 9-54. Oxford University Press.
Bird, Steven, and Jonathan Harrington (2001). Editorial: Speech annotation and corpus tools. Speech Communication, 33(1,2): 1-4.
Bird, Steven, and Mark Liberman (1999). Annotation graphs as a framework for multidimensional linguistic data analysis, Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, ACL, Madrid, Spain. http://acl.ldc.upenn.edu/W/W99/W99-0301.pdf
Bird, Steven, and Mark Liberman (2001). A formal framework for linguistic annotation. Speech Communication, 33(1,2): 23-60.
Bird, Steven, and Gary Simons (2003). Extending Dublin Core Metadata to support the description and discovery of language resources. Computing and the Humanities, 37, 375-388.
Bird, Steven, and Gary Simons (2003). Seven dimensions of portability for language documentation and description. Language, 79(3): 557–82.
Bow, Catherine, Baden Hughes, and Steven Bird (2003). Towards a general model of interlinear text. In Proceedings of the EMELD Conference 2003: Digitizing and annotating texts and field recordings. http://linguistlist.org/emeld/workshop/2003/proceedings03.html.
Brants, Thorsten, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The TIGER Treebank. In E. Hinrichs and K. Simov (eds.), Proceedings of the First Workshop on Treebanks and Linguistic Theories, pp. 24–41, Sozopol, Bulgaria.
Bruce, Gösta (1989). Report from the IPA working group on suprasegmental categories. Working Papers, Lund University, Department of Linguistics, 35: 15-40.
Brugman, H., P. Wittenburg, S. C. Levinson, and S. Kita. Multimodal annotations in gesture and sign language studies. In M. Rodriguez González & C. Paz Suárez Araujo, eds., Third international conference on language resources and evaluation(pp. 176-182). http://www.mpi.nl/institute/research-groups/language-and-cognition-group/publications
Cassidy, Steve, and Jonathan Harrington (2001). Multi-level annotation in the Emu speech database management system. Speech Communication, 33 (1,2): 61-77.
Comrie, Bernard, Martin Haspelmath and Balthasar Bickel. (2004, revised 2008). The Leipzig Glossing Rules. Available at:
http://www.eva.mpg.de/lingua/resources/glossing-rules.php
Edwards, J., & Beckman, M. E. (2008). Methodological questions in studying phonological acquisition. Clinical Linguistics and Phonetics, 22(12): 939-958.
Eisen, B. (1993). Reliability of speech segmentation and labelling at different levels of transcription. Proceedings of the 3rd European Conference on Speech Communication and Technology, Vol. 1, pp. 673-676.
Frank, A. 2000. Automatic F-Structure Annotation of Treebank Trees. In M. Butt and T. H. King (eds.), The Fifth International Conference on Lexical-Functional
Grammar, The University of California at Berkeley, July 19-20 2000, CSLI Publications, Stanford, CA.
Gabbard, Ryan, Seth Kulick, and Marcus Mitchell. 2006. Fully parsing the Penn Treebank. Proceedings of the Main Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, New York, NY, 184–191. Morristown, NJ: Association for Computational Linguistics.
Hajicova, E. 1998. Prague Dependency Treebank: From Analytic to Tectogrammatical Annotation. In Proc. TSD’98.
Krotov, Alexander, Mark Hepple, Robert J. Gaizauskas, and Yorick Wilks. 1998. Compacting the Penn Treebank grammar. Proceedings of COLING/ACL98: Joint Meeting of the 36th Annual Meeting of the ACL and the 17th International Conference on Computational Linguistics, Montréal, Canada, 699–703. Morristown, NJ: Association for Computational Linguistics.
Hewlett, Nigel and Waters, Daphne (2004). Gradient change in the acquisition of phonology. Clinical Linguistics & Phonetics,18 (6): 523-533.
Ide, N. and Suderman, K. (2007). GrAF: A Graph-based Format for Linguistic Annotations. Proceedings of the Linguistic Annotation Workshop, held in conjunction with ACL 2007, Prague, June 28-29, 1-8.
Ide, N., Romary, L. (2007). Towards International Standards for Language Resources. In Dybkjaer, L., Hemsen, H., Minker, W. (Eds.), Evaluation of Text and Speech Systems, Springer, 263-84.
Ide, N., Romary, L.. (2006). Representing Linguistic Corpora and Their Annotations. Proceedings of the Fifth Language Resources and Evaluation Conference (LREC), Genoa, Italy.
Ide, N., Romary, L. (2004). International standard for a linguistic annotation framework. Journal of Natural Language Engineering, 10:3-4, 211-225.
Ide, N., Romary, L. (2004). A Registry of Standard Data Categories for Linguistic Annotation. Proceedings of the Fourth Language Resources and Evaluation Conference (LREC), Lisbon, 135-39.
Ide, N., Bonhomme, P., Romary, L. (2000). XCES: An XML-based Standard for Linguistic Corpora. Proceedings of the Second Language Resources and Evaluation Conference (LREC), Athens, Greece, 825-30.
Jacobson, Michel, Boyd Michailovsky, John B. Lowe (2001). Linguistic documents synchronizing sound and text. Speech Communication, 33 (1,2): 79-96.
Kipp, Michael (2001). Anvil - A Generic Annotation Tool for Multimodal Dialogue Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech), pp. 1367-1370, Aalborg, September 2001.
Lausberg, Hedda, and Han Sloetjes (2009). Coding gestural behavior with the NEUROGES–ELAN system. Behavior Research Methods, 41 (3), 841-849.
Lehmann, Christian (1983). Directions for interlinear morphemic translations. Folia Linguistica 16.193-224.
Marcus, Mitchell, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large annotated corpus of English: The Penn Treebank. Computational Linguistics 19.313–30.
Marcus, Mitchell, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: annotating predicate argument structure. Proceedings of the workshop on Human Language Technology, Princeton, NJ, 110–5. Morristown, NJ: Association for Computational Linguistics.
McKelvie, David, Amy Isard, Andreas Mengel, Morten Baun Møller, Michael Grosse, and Marion Klein (2001). The MATE workbench -- An annotation tool for XML coded speech corpora. Speech Communication, 33 (1,2): 97-112.
Nakatani, Christine H., Barbara J. Grosz, David D. Ahn ,and Julia Hirschberg (1995). Instructions for annotating discourse. TR: 21-95, Harvard University, Cambridge, MA.
Navarro, Borja, Montserrat Civit, M. Antonia Martí, R. Marcos, B. Fernández. 2003. Syntactic, Semantic and Pragmatic Annotation in Cast3LB. Shallow Processing of Large Corpora (SProLaC), a Workshop on Corpus Linguistics, Lancaster, UK.Neidle, Carol, Stan Sclaroff, and Vassilis Athitsos (2001). SignStream: A tool for linguistic and computer vision research on visual-gestural language data. Behavior Research Methods, 33, 311-320.
Pitrelli, John F., Mary E. Beckman, and Julia Hirschberg (1994). Evaluation of prosodic transcription
labelling reliability in the ToBI framework. Proceedings of the 1994 International Conference on Spoken Language Processing, Vol. 1, pp. 123-126.
Pye, C., K. A. Wilcox, and K. A. Siren (1988). Refining transcriptions: the significance of transcriber ‘errors’
Journal of Child Language, 15, 17–37.
Quek, Francis, Dan McNeill, Robert Bryll, and Mary Harper (2002) Gesture Spatialization in Natural Discourse Segmentation. Proceedings of the Seventh International Conference on Spoken Language Processing, Vol. 1, Denver CO, pp.189-192.
Stirling, Lesley, Janet Fletcher, Ilana Mushin, and Roger Wales (2001). Representational issues in annotation: Using the Australian map task corpus to relate prosody and discourse structure. Speech Communication, 33 (1,2): 113-134.
Syrdal, Ann K., Julia Hirschberg, Julie McGory, and Mary Beckman (2001). Speech Communication, 33 (1,2): 135-151.
Taylor, A., Marcus M. and Santorini B. 2001. The Penn TreeBank: an Overview. In Abeillé A. (ed.), Building and Using Syntactically Annotated Corpora, Kluwer.
Taylor, Paul, Alan W. Black, and Richard Caley (2001). Heterogeneous relation graphs as a formalism for representing linguistic information. Speech Communication, 33 (1,2): 153-174.
Telljohann, Heike, Erhard Hinrichs, and Sandra Kübler. 2004. The TüBa-D/Z treebank: Annotating German with a context-free backbone. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004), Lisbon, Portugal, 2004.
Telljohann, Heike, Erhard Hinrichs, Sandra Kuebler, and Heike Zinsmeister. 2006. Stylebook for the Tuebingen Treebank of Written German (TueBa-D/Z). Technischer Bericht, Seminar fuer Sprachwissenschaft, Universitaet Tuebingen, Tuebingen. Revidierte Fassung.
Trippel, Thorsten, Michael Maxwell, Greville Corbett, Cambell Prince, Christopher Manning, Stephen Grimes and Steve Moran (2008). Lexicon Schemas and Related Data Models: when Standards Meet Users. Proceedings of the Sixth International Language Resources and Evaluation Conference (LREC'08). text of paper and slides
Xue, Nianwen, Fu-Dong Chiou, and Martha Palmer. 2002. Building a Large-Scale Annotated Chinese Corpus. In Proceedings of the 19th International Conference on Computational Linguistics (COLING 2002), Taipei, Taiwan.

Group 7: Collaboration Structure

EmilyMBender — Thu, 10 Sep 2009 19:11:42 CDT

The collaboration structure group is charged with considering methods for enhancing collaboration and communication on three levels. Level 1 involves forming communication pathways that can link individual researchers to the overall agenda of developing a shared cyberinfrastructure. Level 2 involves the collaborations that are needed between system developers and tool developers to assure maximum interoperability and open access between data formats and programs. Level 3 involves support for collaborations between linguists and researchers in other sciences, grounded on the use of a shared, open access cyberinfrastructure. For each of these levels, we need to design lightweight methods for ensuring ongoing collaboration and coordination.

The specific agenda items for this group include:
1. Data level interoperability: roundtrips between formats, transductions of formats, funding for the process of developing compatibility
2. Tool level interoperability and methods for collaboration in tool development.
3. Issues arising from a commitment to open access.
4. How to maximize data-sharing: the role of NSF, NIH, and LSA in terms of promoting greater commitment to data-sharing.
5. Characterization of linguistic digital data types and methods for linking to non-digital data.
6. An agenda for developing linkages to other sciences - the bigger picture of Extended Linguistics.
7. Lightweight administration within a framework of complex organizations: NIH, NSF, LSA, CLARIN, etc.

Following are further analyses of these seven agenda items:

1. Data level interoperability.

Level 1 involves compatibility in annotation format, such as a formats provided by frameworks such as Annotation Graphs (AG) or the Linguistic Annotation Framework's Graph Annotation Format (GrAF).
Level 2 involves compatibility in terms of data categories (content), wherein categories are the same conceptually and can be mapped to one another. This can be facilitated by ontological resources like GOLD as well as the ISOcat
Level 3 compatibility involves use of a common set of notational conventions to express a fully declared range of content categories.

2. Tool level interoperability and methods for collaboration in tool development.

LRT, TalkBank, E-Meld standards are largely similar.
Media standardization issues for streaming serving and programs. YouTube, Google, Mozilla, and others are developing standards and systems that we could adopt.
Roundtrips between tool formats: CHAT, Anvil, EAF, AG, EXMaRLDA, Wavesurfer, TEI, SALT. Some of these tools are already interoperable, but these pathways need to be made clearer to users.
AG Tools approach. This approach allows programmers to develop new tools from the AG Toolkit. However, this is only linked at Level 1. Can there be a similar approach at Levels 2 and 3?

3. Issues arising from a commitment to open access. What data must be kept away from the public and what data can be made freely available? How can linguists work together to increase access to larger amounts of linguistically important data?

Sharing and IRB principles at talkbank.org/share.
Legacy data vs. forward-looking protocols (E-Meld, AphasiaBank, as examples)
Community, population constraints

4. Methods for promoting a higher level of data contribution and individual researcher "buy in".

Inducements: publication, easy tool linkage
Community: role of LSA
Obligations and standards: role of NIH, NSF, DARPA, IE
Leading role of the European Community in setting standards for data-sharing for grant recipients

5. Characterization of linguistic digital data types and methods for linking to non-digital data.
Here, it is important to distinguish the emphasis on corpora and linked media from the many other types of digital data that are of interest to linguists. In the area of Linguistic Exploration, the fundamental objects may be word lists, sentence lists, or dictionaries. In Linguistic Anthropology digitized records of objects are important. This extends eventually over to Archaeology and even information on human genetics etc. In the Learning Sciences, there is an emphasis on linking classroom video to individual student portfolios that may include letters, tests, art work and so on. For digital libraries, it is important to make clear where the hard copies actually reside. For many of these objects, identification can be made through the assigment of digital object identifiers (DOIs). However, this is a largely unexplored territory for most linguists.

6. An agenda for developing linkages to other sciences - the bigger picture of Extended Linguistics. Here, the MacWhinney-Groves NSF report should be particularly helpful.

7. Lightweight administration within a framework of complex organizations: NIH, NSF, LSA, CLARIN, etc. There is a perception that some work on the development of shared cyberinfrastrucure has been top-heavy on committee work and reports without producing a significant amount of shared interoperable resources. Is there a way to build organizational structures that produce open-access products? Who should determine patterns of collaboration or should these patterns "emerge" through specific less-organized exchanges. But then how these interactions be guided toward cooperation and interoperability? Perhaps an emphasis on standards for collaboration might be possible.

Recommended Readings

SIGAnn website
TalkBank website (see ground rules for sharing, software, browsable database)
NSF SBE Cyberinfrastructure Report
SILT Proposal (attachment)
FLaReNet website
ISO committee for Language Resource Management webpage
Linguistic Annotation Framework and Graph Annotation Format descriptions (attachments)

Additional Links

ISO TC37 SC4 - Language Resource Management : Homepage
- ISO Linguistic Annotation Framework (LAF) Overview and Draft Standard (under revision); XML Serialization for LAF : ISO Graph Annotation Format (GrAF)

ISO Morphosyntactic Annotation Format (MAF)
ISO Syntactic Annotation Format (SynAF)
ISO Semantic Annotation Format - Time and Events (SemAF-TIME) (formerly TimeML)
ISO Lexical Markup Framework (LMF): Homepage (with Publications and Tools)
ISO Data Category Registry for linguistic concepts (ISOcat)

White paper draft/notes

Introduction

In considering collaboration, we distinguish first between joint research efforts and large-scale coordination. The former may cross many boundaries: of disciplines, institutions, and nations. It is also by and large already successful, and not something that we thought this group needed to focus directly on (though of course a cyberinfrastructure for linguistics would, as an important contribution, provide further support for this kind of collaboration). Rather, we defined our task as the fostering of large-scale coordination such as is required to get the field of linguistics (and sister disciplines in the language sciences) to organize around the creation of a cyberinfrastructure and its population with data. We saw three main facets to the large-scale coordination of effort: cooperation, community building, and communication. The following figure summarizes the relationships we see among these concepts.

Large-scale coordination of effort

Under the heading of large-scale coordination of effort, we include such things as the reuse of data, especially across (otherwise) unrelated research groups; mechanisms for publishing/sharing data; evaluation/quality control of resources (broadly defined to include data sets, tools, standards, etc); establishment of and agreement on standards; and coherence of principles, goals, and architectures across groups involved in the creation of a cyberinfrastructure. In all of these concerns, we place a high priority on avoiding duplication of effort. In order to foster large-scale coordination of effort, we see three main classes of tools: formal modes of cooperation, explicit work towards community building, and vehicles of communication.

Coordination Models

We identified a few examples of models of coordination:

ISO: exploit existing infrastructure for coordination
TalkBank: coordinate among researcher groups such as AphasiaBank, CHILDES, PhonBank, ClassBank, etc.
TEI: build infrastructure from the ground up
CLARIN: exploit an EC framework (ESFRI) financing the feasibility part of establishing European infrastructures, for CLARIN in the Humanities, to be then continued with national funding from the various EU countries
FLaReNet/SILT: parallel international funded efforts to establish international networks, with bottom up coordination by the projects' coordinators

Clearly, the various coordination models are linked to different funding models, and can (to differing degrees) be deliberately fostered by funding agencies.

The Role of the LSA

An informal working group has been created within the LSA, consisting of LSA staff, leadership and technology consultants. The working group will make detailed recommendations to the LSA's Executive Committee concerning specific actions the LSA can undertake, both immediately and in the future, to facilitate the development of a cyberinfrastcuture for linguistics, disseminate information related to this endeavor, and promulgate a "culture change" with regard to sharing of data, tools, and results. These actions would build on the initial steps already taken by the LSA in this regard, such as its digital publishing platform, eLanguage.

Community Building

We began with the observation that successful technology is always supported by (while also supporting) a community of users. We define a community for these purposes as a group of people working on similar issues, using the same tools/platforms/resources, who talk to each other and who share principles and practices in their research efforts. Different communities may be display these various properties to differing extents. To give some examples, SIGs (special interest groups within larger scholarly bodies) illustrate communities primarily defined by working on similar issues. Computational linguistic communities that have grown up around the creation of use of tools and resources include the developer and user groups of WordNet, FrameNet, NLTK, GATE and UIMA. There are also communities of linguists who are producing shared databases, but not new computational tools or formats. These include groups such as CHILDES, PhonBank, AphasiaBank, LIDES, etc. Vehicles for communication (discussed further below) can also create communities of their own. Examples here include the readership of LINGUISTList and the readership of the Corpora list. Note also that communities can vary greatly in size (readership of LINGUISTList being at one extreme) and of course overlap with one another, as individuals belong to multiple different communities.

Since communities are important for both communication (see below) and the success of software projects, we considered ways to foster the development of communities, while recognizing that such things cannot be precisely engineered (nor their success precisely predicted). Means of community building include:

Funding programs sponsoring multiple groups working on the same/similar problems (e.g., language documentation sponsored by ELDP, DARPA programs)
Evaluation campaigns (lots of examples from compling here: Semeval, MUC, TREC, CLEF, CoNLL shared tasks)
On-line fora (from big like LINGUIST List to small like user groups for particular tools)
SIGs (of particular relevance here is ACL's SIGANN)
Workshops (e.g., LAW, E-MELD workshops, Cyberling 2009) and conferences (LREC is of particular note here in having created a community)
Tutorials
- At summer schools (LSA institute, EuroLAN, Johns Hopkins, ESSLLI)
- On the web
- As part of funded projects (e.g., SILT)
Journals (LRE, Language, eLanguage, Computational Linguistics...)

Communication

We identified communication as a problem that cross-cuts many aspect of large-scale coordination of effort. In particular, we need to communicate about standards (availability and development), tool and resource availability, needs assessment, and principles & practices. People working on tools and standards across linguistics and the language sciences more broadly need to be aware of each other and each other's efforts and need to be able to communicate with potential users for needs assessment. (Mark Liberman commented that every successful piece of software starts with someone scratching an itch: they have a problem, build a solution, and share that solution. But not everyone who has a problem that can be solved with software has the means/skills to build that software themselves.) People potentially using tools and standards need to be able to find them. People who should be using tools and standards but don't yet know about them, need to be reached.

With these kinds of communication in mind, we developed a list of potential communication vehicles:

Existing communities' infrastructure (newsletters, meetings, websites), including both informal communities and scholarly organizations
Teaching materials, especially made available over the web (syllabi, problem sets, web-based tutorials)
Wikis/blogs ("bottom-up web-based communication")
- On-going maintenance of information collections
- Reasons for people to come back to the on-line communication site
- ad words: Set up context sensitive "ads" on the model of Google AdWords which could run on LINGUIST, lsadc.org, etc, where the things being advertised are relevant projects and standards (and no money is exchanged)
Funded collaborations (e.g., SILT/FLaReNet)
Workshops/tutorials
Reviewing guidelines/review feedback
- Pushing funding agencies to require plans (and follow through) for using standards and publishing data for proposals that use tools/create data
- Pushing funding agencies to require proposals for new tools/standards to appropriately cite and situate themselves within the existing tools/standards ecology
- Conference/journal reviewing check for appropriate citations of data, tools, resources
Resource maps/eliciting metadata (cf. LREC 2010)
Journals like Journal of Experimental Linguistics which publish code along with the resulting research.
Idea from Steve Moran: A new journal (perhaps in the eLanguage set) on the model of Journal of Experimental Linguistics, which publishes data sets collected in the field. Maybe called "Journal of Linguistic Description"?

Content

This section briefly outlines the content that we need to be communicating about within the large-scale coordination of effort required to bring about a cyberinfrastructure for linguistics (and the language sciences).

Data level interoperability, on two levels: Interoperability of data and annotation format, including means of mapping between existing formats (cf. GrAF) and interoperability of annotation content, via ontologies (cf. GOLD) or other inventories of linguistic categories (cf. ISOcat).
Tool level interoperability and methods for collaboration in tool development.
Managing issues arising from commitment to Open Access, establishing and publicizing shared principles.
Promoting individual researcher "buy-in". A cyberinfrastructure is not useful until it is populated with data, but our field is in need of culture change in this respect. We believe this culture change can be achieved through a combination of making it easier for individual researchers to contribute data (through useful tools), educational campaigns on the part of the LSA and similar groups, funding agencies establishing policies requiring data sharing, and publication venues requiring testing against and citing existing available data.
Connection to other fields: Linguistics is only one of the language sciences, and digitized data from many fields (education, political science, law, ...) can be valuable for linguists. As we develop our infrastructure, we need to be mindful of how it fits into this larger ecology, and where teaming up with other language sciences can bring economies of scale. Closer to home, the field of computational linguistics has a good deal of cyberinfrastructure (and communication around cyberinfrastructure) established. We envision creating a portal into cyberinfrastructure concerns for people who identify as linguists, which rather than attempting to encompass all language-related cyberinfrastructure itself links to existing efforts in allied disciplines.

Summary: 5 Cs of cyberinfrastructure

Collaboration: Joint research alone isn't enough to bring about a cyberinfrastructure, though it will play a key role. We need large-scale coordination of effort.
Cooperation models: There are many ways to coordinate effort, and we will probably use all of them.
Coordination: On the technical side, we need interoperability, which entails coordination on standards
Communication: On the people side, we can't achieve coordination without communication, bringing people in, making them aware of each other, and keeping them in touch.
Community Building: Key to both successful communication and successful software.

Action items - Short term

Draft recommendations to funding agencies regarding standards, data publication, etc.
Draft recommendations to journal editors and conference organizers regarding citing tools/resources and publishing data
Create teaching resources (through LSA?)
Continue this conversation (all WGs, on the wiki for now)

Action items - Long term

Ensure communication among projects/groups
Push those developing standards to for specific areas (e.g., PHON group connected to TalkBank) to contribute to ISO TC37 SC4
Work towards data/annotation harmonization

annotation streams

mebeckman — Tue, 08 Sep 2009 09:20:41 CDT

This wiki page is a record of a discussion about terminology. The Annotation Standards working group considered the different terms being used for different paths (the logical term if we use the annotation graph representation), and for describing various types of relationship among different paths. For now this page is simply the body of e-mail messages in this thread, which could be edited later to make a coherent description of the terminological ambiguities, relating them to discussion of the ontology in the paragraph where the "white paper" cites the Annotation Graph framework of Bird and Liberman (2001), and linked into the main page there if/when these pages migrate to their eventual home.

n.b. Calling the paths "streams" invokes Hertz's Delta system, as described in, e.g.:
Hertz, Susan R. (1990). The Delta programming language: an integrated approach to non-linear phonology, phonetics, and speech synthesis. In John Kingston & Mary E. Beckman (eds.), Papers in Laboratory Phonology I: Between the Grammar and Physics of Speech, pp. 215-25. Cambridge University Press.

_____________________________________________________________________

From: Charles Fillmore
Subject: "layered annotation"
Date: Sun, 6 Sep 2009 16:58:45 -0700

I think I might have introduced a confusion of terminology in connection with "layered annotation."

Since we're in general talking about stand-off annotations, we can have one layer that shows part of speech, one that shows lexical meanings, one that shows speaker switch, etc., etc., and that's all fine. But I find that I've also used "layering" to refer to the problem of using one annotator's product to form the basis of another level of analysis (and have claimed that this is both necessary and problematic). Can anybody think of a way of phrasing the latter that would avoid confusion?

From: Sarah Churng
Date: Sun, 6 Sep 2009 17:20:23 -0700

For 'layers' aimed at the simultaneous displays of different time-aligned annotations such as pos tags, lexical transcriptions, intonational phrases, etc. (in contrast to the levels of analysis 'layers'), I think we referred to these in our meetings as tiers for linguistic types?

In ELAN, stackable tiers are available both as independent tiers which are time-alignable (such as direct transcriptions) and as referring tiers which are not time-alignable and must inherit from parent transcription tiers (such as with translations).

Is this getting close to what you mean?

From: Stuart Robinson
Date: Mon, 7 Sep 2009 00:36:54 +0000 (UTC)

At the risk of muddying the waters, a tier in Sarah's sense could have multiple layers then, no? So Chuck's example would be one where a tier for morphosyntax or whatever has multiple layers (i.e., annotation on annotation). On top of that you might also have versioning.

From: Sarah Churng
Date: Sun, 6 Sep 2009 18:41:04 -0700

Right, and, to a large extent, I think the documentation for ELAN actually faces the same conundrum Chuck brings up---that is, they seem to use "tier" in both senses Chuck is trying to distinguish. First, there are the tiers I mention below for aligning "layers" of POS tagging, lexical meaning, etc, and these straightforwardly can be handled across a time axis.

Second, there are tier-to-tier relationships, so to speak, between different tier types. These are of the kind that Chuck was originally asking to give a different label than "layer" to in his e-mail. ELAN gets around this by distinguishing "parent" vs. "child" tiers. So, the product of a parent tier is available for the annotation of a child tier, but not vice versa.

Is this something we can adopt? It nicely admits that there is an overlap of "layers" for both senses of the word, but makes it clear that not all layers are created equal in the sense of what each annotation can inherit. And the issue of multiple embedded layers that Stuart brings up is independent and able to coinhabit with the different 'parent' vs. 'child' layers.

The documentation on tiers in ELAN: http://www.lat-mpi.eu/tools/elan/manual/ch05s01.html/view

From: Mary Beckman <mbeckman@ling.osu.edu>
Date: Mon, 7 Sep 2009 09:41:19 -0400

Muddying the waters further (or perhaps clarifying?), I think the distinction that we're making maybe is not between types of annotation streams and their relationships, but between a static and a dynamic view, no?

That is, when I hear/read "tier", I tend to think of the simultaneous display and/or the associated simultaneous development of different streams of parallel annotations that are anchored to the same primary data. This anchoring can be either via a time stamp (as in Figure 1) or just via reference to shared nodes in the annotation graph (as in the phrase-internal sharing of nodes for the word-by-by transcription and gloss in Figure 2). But this is a static "result-oriented" view of the parallel analysis streams and their relationships.

By contrast, when I hear or read "layer", I tend to think of the dynamics of how the different annotation streams were originally developed. Although there are cases where the analyses have a necessary order -- e.g., a word-by-word gloss probably has to come after a transcription and tokenization of the primary data -- there are also many cases where the "layering" is arbitrary or idiosyncratic. For example, in (ame_)ToBI labelling, some people, such as Stef Shattuck-Hufnagel, find that they have to mark the Break Indices first and then go back and mark the Tones. Others, such as Nanette Veilleux (and me), can't do Break Indices before we do Tones. If the annotation isn't left in a partial state, though, there is no way to recover the difference between Stef and Nanette. So this is a dynamic "process-oriented" view of the relationship.

Machine reusability of data

alexispalmer — Tue, 08 Sep 2009 06:27:52 CDT

NOTE: this page is still in process. at present it contains a number of conceptual recommendations without suggestions of how these recommendations could be implemented. Please feel free to contact the author with comments, complaints, and suggestions: Alexis Palmer, apalmer@coli.uni-sb.de

This page discusses one particular manner in which linguistic data may be re-purposed: as training data for statistical machine learning approaches in computational linguistics and/or natural language processing. More specifically, we're talking about training data for supervised or semi-supervised methods -- methods that learn from labeled data.

Linguistic data as training material for machine learning

Case study subdiscipline:	Computational linguistics, language documentation and description
Goals of this case study:	To highlight how decisions related to the annotation and storage of linguistic data (in this case, interlinear glossed texts from a language documentation project) can make the data more or less useful as training data for statistical machine learning methods

1. What makes data good training data?

The key word here is consistency. Roughly put, the machine model learns generalizations over observed data and uses those to predict analyses for previously-unseen data. In order for it generalize well, the collection of data must be as internally-consistent as possible in the way that it is coded/labeled.

A second important consideration is the underlying data structure. For efficient machine processing, there must be some explicit indication of relations between the text and its annotations. These two points are illustrated below with examples of interlinear glossed text, a common way of representing language data.

2. Labeling consistency

Typographic consistency
It is important in any data collection and annotation effort that all annotators work from one agreed-upon set of labels. For the sake of the machine learner, it is also important to adhere to capitalization and punctuation conventions. For example, 'PST' and 'pst' may both be intended to indicate a past tense morpheme, but the machine will see them as two distinct labels. Of course, many such issues can be handled by processing the data post-annotation and pre-model training, but to do so efficiently requires text manipulation skills that those producing the original data may or may not have.

One way for projects to maintain labeling consistency is by use of an annotation interface which restricts the space of allowed labels.

Analytic consistency
Maintaining analytic consistency is a much more difficult task. In cases where the analysis is reasonably well-understood at the outset of annotation, agreed-upon conventions for analysis and annotation may be made available to annotators in the form of a detailed annotation manual. It is often the case, however, that analysis and annotation proceed in parallel. In documentation and description of less-studied (or previously unstudied) languages, this is in fact the normal situation.

Several bits of record-keeping can help to deal with changing analyses:

tracking the source of each label (i.e. the specific annotator) as well as the time and date of annotation
documenting changes in analysis and/or labeling conventions, indicating the nature and source of the change, how the change should be manifested in the annotation (in other words, what did the previous analysis look like? what does the new analysis look like?), the date and time at which the decision to change the analysis was made, and whether or not the change has been back-propagated to previously-labeled data
using annotation tools and/or data formats which are able to maintain a historical record of changes in the data (along with the metadata associated with those changes)

We recognize that some of these desiderata are not easily attainable with currently-available systems for text glossing and interlinearization, particularly in the language documentation context. We thus add our voice to those calling for development of an open source, updated, general-purpose system for text interlinearization and glossing.

3. Data structures

First, we point to the pages of WG1: Annotation Standards as well as the Existing Resources page for many valuable resources pertaining to standardization of data structures for annotation. The resources presented on these pages include links to proposed standards and extensive bibliographic references related to this topic.

Interlinear glossed text (IGT)
The particular concern in this case study is the use of interlinear glossed text (IGT) as training data for a machine learner. First, here's an example of IGT from the Mayan language Uspanteko (Pixabaj et al.).

Full text: Kita' tinch'ab'ej laj inyolj iin.

TEXT	kita'	tinch'ab'ej				laj	inyolj		iin
MORPHEME	kita'	t-	in-	ch'abe	-j	laj	in-	yolj	iin
GLOSS	NEG	INC-	E1S-	hablar	-SC	PREP	E1S-	idioma	yo

Spanish translation: No le hablo in mi idioma.
English translation: I don't speak to him in my language.

Links between annotation tiers
The table above shows three tiers of annotation for this Uspanteko clause. The 'TEXT' tier contains each word of the clause (word boundaries are indicated by double-line cell borders). The 'MORPHEME' tier shows a segmentation of each word into its component morphemes, and the 'GLOSS' tier shows a morpheme-by-morpheme gloss of the clause, including both gloss labels for non-stem morphemes (e.g. NEG for kita') and lemma translations for stem morphemes (e.g. hablar for ch'abe).

Two NLP tasks we might imagine learning from such data are morphological segmentation (producing the 'MORPHEME' tier, given at least the 'TEXT' tier and perhaps the translation(s) as well) and morpheme glossing (roughly, given the 'MORPHEME' tier, produce the 'GLOSS' tier). This is where the data structure used to represent the interlinear text becomes crucial!

Most often when we encounter IGT -- as, in fact, in the table above -- the links between annotation tiers are conveyed through visual aspects of the presentation. Here, for example, the association of morphemes with the words they belong to is communicated using double-line borders at word boundaries. Visually-oriented presentations of IGT do not generally provide the explicit encoding of these relationships that a machine learner needs to make sense of the data. In order to use IGT as training data, it must be presented to the machine learner in a format that directly encodes links between elements from one annotation tier to those on another.

Structured representational formats
What is needed to address this concern is a format which preserves structured links between annotation tiers. XML formats are one way of preserving said links. At the same time, using XML follows current recommendations regarding longevity and portability of data (for example, Bird and Simons 2003, EMELD School of Best Practices). Several XML formats for IGT have been proposed, including EthnoER's EOPAS (Schroeter and Thieberger 2006), IGT-XML (Palmer and Erk 2007), and an earlier model outlined in Bow, Hughes, and Bird 2003 (Bow et al. 2003). Another approach is the use of Annotation Graphs (e.g. Bird and Liberman 2001, Maeda et al. 2002).

References:

---Bird, Steven and Mark Liberman. 2001. 'A formal framework for linguistic annotation.' Speech Communication. 33(1-2): 23-60.
---Bird, Steven and Gary Simons. 2003. 'Seven dimensions of portability for language documentation and description.' Language, 79(3): 557-582.
---Bow, Catherine, Baden Hughes, and Steven Bird. 2003. 'Towards a general model of interlinear text.' In Proceedings of EMELD Workshop 2003: Digitizing and Annotating Texts and Field Recordings. LSA Institute: Lansing MI, USA.
---Maeda, Kazuaki, Steven Bird, Xiaoyi Ma, and Haejoong Lee. 'Creating Annotation Tools with the Annotation Graph Toolkit'. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC).
---Palmer, Alexis and Katrin Erk. 2007. 'IGT-XML: An XML format for interlinearized glossed text.' In Proceedings of the Linguistic Annotation Workshop (LAW-07), ACL 2007.
---Pixabaj, Telma Can (coordinator), Miguel Angel Vicente Méndez, María Vicente Méndez, and Oswaldo Ajcot Damián. Uspanteko text collection, in Text Collections in Four Mayan Languages, 2003-2007. OKMA (Oxlajuuj Keej Maya' Ajtz'iib'), Supported by Endangered Languages Documentation Programme (SOAS, University of London).
---Schroeter, Ronald and Nicholas Thieberger. 2006. 'EOPAS: the EthnoER online representation of interlinear text.' In Sustainable Data from Digital Fieldwork (proceedings of conference held at the University of Sydney, 4-6 December 2006). Sydney University Press.

Discussion: Where should all this information go?

mebeckman — Mon, 07 Sep 2009 04:53:29 CDT

HThere was some discussion at the workshop of eventually moving some or all of the information on this wiki into a wikipedia article or series of articles. This page exists to host a discussion of that idea, and provide a forum for other ideas about where this information should live going forward.

Wikipedia and/or glottopedia

Martin Haspelmath suggested that, while wikipedia may be appropriate for disseminating articles aimed at the proverbial "educated lay reader", we need to also disseminate more technical information to linguists using a specialist site such as glottopedia. Mark Liberman countered that much information on wikipedia is quite technical, citing the articles on various terms and concepts in maths and computer science, such as this article on maximum likelihood or this article on ontology (in information science).

In evaluating the arguments for and against establishing a separate wiki for more technical articles, we probably should look at opinion pieces such as this article on why does wikipedia suck on science, where Thomas Goetz says, "Curious about just what epigenetics is? Figure you really should know what mitochondria do? Don’t count on Wikipedia - odds are their analysis is too pedantic for you, as it is for me."

In deciding whether and how to use Wikipedia, we may want to look at an initiative at NIH to train NIH scientists in how to write effective Wikipedia articles. Wired Science has an article about a "Wikipedia Academy" that NIH hosted recently.

It may also be useful to review some of the insights about the sociology of wiki culture in Marshall Poe's article in the September 2006 Atlantic Monthly and Nicholson Baker's March 2008 New York Review of Books piece. More information on these issues can gleaned from reading the opinions expressed in (and experiences with Wiki editing recounted in) some of the comments on this Ars Technica piece on the Maurice Jarre Wikipedia hoax.

Group 4 White Paper

haspelmath — Wed, 02 Sep 2009 09:48:50 CDT

Data Reliability and Provenance

Peter Austin (co-chair)
Martin Haspelmath
Kurt Bollacker
Tracy Holloway King (co-chair)
Koenraad de Smedt
Paul Trilsbeek

Abstract

In this paper we discuss what data provenance and data reliability are, with special attention to the needs of the linguistics community when moving from simple data sets to more complex cyberinfrastructures. The integrity and completeness of provenance information as a basis for citation, rights management, etc. is crucial for those who record, annotate and compile materials, but also for the sources of raw materials, in particular for indigenous and minority communities, and for the scholars who use these materials as a basis for further scientific work. We suggest some first steps to promote data sharing and publication in the linguistics community.

Data Provenance

Provenance is the who, what, and when of metadata. When a data set is created, it is important to know where it comes from and who is responsible for its publication. Adequate information about provenance, i.e. information about how, where, when and by whom the data was collected, encoded and annotated, and who assumes responsibility for its publication, allows the quality of the data set to be assessed, provides a contact in case of questions, and establishes authorship of the data set, similar to the authorship of an academic paper.

The contents and status of a data set are not always clear from a quick inspection of the data. The data might come from native-speaker informants, from published works of literature, from the web, etc. Adequate metadata are needed for cataloguing data sets so as to make them searchable, and to secure that the data will be used in an appropriate way. Metadata should include a detailed description of the data set, including sufficient information on its provenance. In some cases provenance information should even be provided for separate sections of the data set wherever differences are relevant. For example, the data may have been collected over several years of field work; knowing when each section was collected and by who and under which circumstances may be important.

Provenance is extremely important to the scientific community which uses data sets as the basis of further analysis, hypothesis testing, etc. For the user, provenance is crucial assessing whether, and to what extent, a data set can form an appropriate basis for subsequent scientific work. Knowing how the data was collected and processed and who was responsible for it allows future users of the data set to judge the quality of the data, how it fits in with their research and how it compares to other data sets. Provenance is also important for the replicability of data and research results. Unless it is clear where the data came from and how it was created, data sets cannot be replicated. For example, if a data set has constituency trees annotated over it, it is important to know whether the trees were manually constructed, created automatically, or bootstrapped by manually correcting automatically constructed trees.

For the creators, provenance provides a way to get credit for the scientific work done in creating the data sets and allows the community to cite the data sets with their authors in academic works. Creating a high quality data set is extremely time consuming and requires highly skilled linguists: as such, it is important that those doing this work get credit for their contribution. Provenance is also important in establishing and maintaining privacy rights for those who provided the data. There are many reasons that data may not be appropriate to publish freely in its entirety, e.g. individuals may be identifiable from videos or from the content of discussions, or private rituals may be recorded. In particular the rights of indigenous and minority communities should be respected and acknowledged.

Provenance tightly linked to the proper, standardized use of metadata and documentation in reliable environments. All data must be tagged with the appropriate metadata and linked to its documentation. This allows researchers to understand what is, and is not, included in the data set and its annotation and to correctly cite the data set, thereby acknowledging the work of the data set creators and allowing future researchers to reproduce their findings. Metadata and documentation is facilitated by adherence to standards and best practices. As students may not be familiar with these, more senior researchers and experts in natural language data curation need to share their knowledge and to facilitate students adherence to the accepted standards.

Achieving Reliable Provenance Through Publication

A major question is how to achieve reliable data provenance in the linguistic community and promoting the sharing of data. Creating data sets is a time consuming, highly skilled task and the scientific community as a whole needs to acknowledge this and to provide support for those working on creating and maintaining data sets. Individuals contributing to and creating these data sets need to get institutional credit for data publication. For example, these should count for tenure reviews and other review processes and should be an integral part of grant applications, both in having data set publication be part of the grant work and in the granting agencies favoring researchers who publish curated data sets, just as they favor those with a proven track record of traditional academic publications.

Given the nature of the linguistics community and the working paradigms that they are used to, we suggest promoting curated data sets as publications. The technology is currently available to treat curated data as publication. There is extensive archival work on linguistic data sets, including the work done by organizations such as the Linguistic Data Consortium. There are also examples from other scientific fields where publication of data collection has become an established scientific practice. However, there needs to be extensive institutional and social engagement in order for curated data as publication to become the norm in the linguistics and language studies communities.

Researchers need to be encouraged to publish curated data sets. This can be done in part by requiring it at the institutional level or as part of a grant reward. It can also be aided by providing more infrastructure to publish data sets, including providing information on best practices and on how to access the necessary technologies. In addition, researchers need to cite the published data sets that they used in their research. Reviewers and publishers of articles and books should reject submissions if they do not cite the data sources they they used in their research. These citations should be standardized by using the metadata and the publishers of the curated data sets should facilitate this by providing information as to how the data set should be cited.

The linguistic community will also need to provide support to ensure annotation and data quality control, just as it does with published academic papers. Not all data collection and annotation is done equally well and is of equal value. The community needs to have a way to recognize and acknowledge this, similar to the relative value of different books publishers, journals, and conference proceedings.

Publication of data through a recognized publication channel with an ISSN would make it easier to cite the data uniformly and correctly, would clearly establish authorship, would allow for different editions, would enforce the use of standards and metadata and would a catalyst for giving academic credit to the makers. It would also promote reviews and rating systems.

In a different field, the data journal Earth System Science Data promotes the rapid publication of research on original data sets. Their policy is outlined as follows:

"Articles in the data section may pertain to the planning, instrumentation and execution of experiments or collection of data. Any interpretation of data is outside the scope of regular articles."

"In the first stage, papers that pass a rapid access peer-review are immediately published on theEarth System Science Data Discussions (ESSDD) website. They are then subject to Interactive Public Discussion, during which the referees' comments (anonymous or attributed), additional short comments by other members of the scientific community (attributed) and the authors' replies are also published inESSDD. In the second stage, the peer-review process is completed and, if accepted, the final revised papers are published inESSD. To ensure publication precedence for authors, and to provide a lasting record of scientific discussion,ESSDD andESSD are both ISSN-registered, permanently archived and fully citable."

It should be noted that articles in ESSD seem to contain the data mostly within the articles themselves. A journal is, however, not the same as a cyberinfrastructure. A data publication is meant to announce to the community that data has been available and how it was obtained and annotated, and also to give credit where it is due. The actual data can be accessed in a variety of ways, not necessarily through the same channel. Since publication is by nature public, there may be some issues with restricted data. Full metadata could be published, but the metadata could stipulate restrictions on the accessibility of the actual data (e.g. due to proprietary data or privacy issues).

One could imagine organizations acting as the publishers of articles with their data sets. Publishers of data will have the responsibility of checking at least the formal aspects of published data, such as proper use of metadata, adherence to standards etc. It might be possible that the same data is published at different hosts. We would need buy-in from institutions and linguists to support this. Publishers would themselves be rated and would need to actively advertise their data publications and make them attractive to researchers. Peer review of data publications should be stimulated. Perhaps language resources would be published unedited first, then reviews can be added on later, where annotation might count as a type of review; by having the language data out, even in unedited form, people would be encouraged to annotate it. Many academic credit systems (e.g. in Norway) require full peer reviewing as well as the use of recognized academic publishing channels.

Persistence and Fine-Grained Provenance Information

Given that linguists change institutions and that URLs shift over time, it is important that future researchers be able to access the same data that is being used today and to be certain that this is the same data as was used by other researchers. On the Internet, the assignment of provenance information to a piece of information can be assured through the use of a Persistent Identifier (PID). PIDs are globally unique identifiers that remain the same even if the URL of a resource changes. Central PID resolvers are used to administer the locations and additional metadata information of the resources. One PID can be used to refer to multiple identical copies of a resource in different locations. Examples of PID systems are the Handle system and the DOI system. PID systems only work though as long as the administration of the resource links is maintained, so the assignment of PIDs alone does not guarantee the long-term stability of the resource references. The linguistic communicty might consider setting up a registration authority for linguistic data.

Every entity involved in data set creation can be identified by a unique PID. These include entities such as people, organizations, and their roles, the data sets and documents themselves, and different views and mashups of the data. Added value by annotations etc. of existing material can be managed by cascading PIDs with different rights. A proliferation of provenance information may increase the size of the information by orders of magnitude but disk space is cheap and so we can do this very fine grained, e.g. sound clips indicated by start and stop time in a speech corpus. Assigning full provenance information can however be complicated, e.g. when material is translated or when value is added to copyrighted material (e.g. Wall Street Journal corpus) or when a speech corpus has a radio broadcast in the background.

Tomorrow's cyberinfrastructures will not be limited to static storage of data, but will be dynamic systems that process and present data according to user's needs. This context will present special challenges to handling provenance information. The following dynamic functionalities can be considered:

customized presentation of data: filtering, reformatting, style sheets
pipelined processes
mashups: the integration of data from various sources (sometimes combining various modes)

In contrast to paper materials, which are static and pre-edited, a cyberarchive allows (and should allow) the user to participate in filtering and presenting information ("play editor yourself"). An example is the Wittgenstein archives at Bergen, which contains digitized manuscripts: the user can choose to include or exclude certain pieces of information and has options for visualization. This creates challenges for provenance. How would you cite a specific view among many other possible views of this material? Such reference may be possible by generating a unique URI and PID for the transformed and formatted web page. An "I want to cite this" button would make this process easy for the user. Dynamic customized views could help with scientific collaboration since it is a good way to bring things together at the presentation level; the tools for this are very useful since most linguists do not have the user interface design skills to do this themselves.

The annotation layered on curated data is often done in conjunction with specific software. Having "software as a service" available to the linguistics community can aid in this process. It will be particularly valuable for institutions with less extensive computing infrastructure, allowing their researchers access to state of the art data set curation facilities. New frameworks such as UIMA can aid in the interactive pipelining of processes on data. This again is a challenge for provenance information. Ideally, a PID can be assigned to every step in the pipelining process; note that at every step, intermediate data could be worth storing as a new resource.

Furthermore, Rosetta, Freebase, the Internet Archive, etc. allow for mashups of data. Cyberinfrastructures should provide mashup functionality, i.e. the smart combination of data from various sources. Provenance is a challenging for mashup data since the mashup process is dynamic and every combination of specific versions of data produces a new mashup version. Also, some people might want to reference the particular rendering of a mashup. Referencing views could easily escalate when every user can have a personal view.

Reliable Identification, Authorization and Rights Management

As part of provenance, in recording the who, what, and when of metadata, it is necessary to have trusted identification of individuals, organizations, and services. This identification needs to persist over time so that decades after a data set is created, it is still possible to determine who created it and how it was created. Reliable identification of individuals is an important prerequisite for at least two purposes:

Identifying authors and sources to give credit to data creation.
Identifying users to provide authorization based on licenses and rights.

CLARIN has stated that some system with global e-identities can solve many problems associated with the current situation of people having different usernames at the various sites they use. A single logon should identify the user in an easy way. This is a largely solved problem on technical level, but various solutions are available and none are widely used by the community:

OpenID
Federations of local e-identity providers

Authorization is a tuple linking a set of rights, the identity of a piece of data and the identity of a user. Authorization can take various forms depending on restrictions, e.g. access can be given to anyone, can be based on email domains (i.e. affiliated institutions), can require the acceptance of a user license, etc. dependent on what the author or stakeholder decides. There are many such restrictions, e.g., access may be given for non-commercial purposes only, or access to sacred songs can be restricted to the initiated, or subparts of data may be proprietary and requires additional agreements. There is a need to train people on what types of rights and access are appropriate. There may be privacy issues with sources such as hospital patient data, sign language data, etc. which may place restrictions on availability.

Some language data, especially spontaneous speech and sign language, cannot be distributed due to privacy issues, in particular in utterances referring to people. becomes very complex with international access, where different countries may have different rules for guarding privacy. This situation may require different country-specific licenses, so international cooperation may need legal advice from the start.

Some technology exists to anonymize source materials, e.g. by masking, manual and automatic, e.g. making non-words out of words, but keeping POS; media is harder to deal with than strict text, especially sign language where need much of the exact original data; any encoding of a facial expression is going to be on the edge. Perhaps 3D models or avatars could be useful, but often not automatic and sometimes not possible, e.g. in sign language.

At the start of the DOBES program, legal specialists advised keeping all data closed. Since that was not an option, a code of conduct was worked out which provides a workable solution. Yet, some data may need to remain closed to all but core researchers in a tightly controlled project. One could use information from original institution waivers to guide what permissions to assign to the data. The CLARIN project has a working group on IPR and licensing issues. The European parliament has expressed an interest in solving the complexity of the issues and may want to promote a revision of legislation, hopefully leading to wider availability of data for research purposes. Some lobbying towards legislators may be necessary.

Researchers are often unwilling to turn over their data for storage and distribution in repositories. One reason is that some people feel their data is not ready yet: once data is in a repository they feel it is cast in stone, which is a real problem for data which is never quite complete such as a dictionary of a living language. It is therefore crucial that repositories offer versioning and updating of the stored materials. Some researchers might prefer to control distribution themselves from their own homepage. A possible solution could be that links or web pages could be generated from the repository automatically. This could also be useful for university administration which has to document research production. A soft approach to repositories could help lead people to understand what a data repository is and what it can start to enable.

A special situation may occur when someone data is retracted, either for privacy reasons or because of errors or other reasons. One could mark data as deleted, invalid or superseded without actually destroying the data; one may also want to be able to temporarily restrict data for certain reasons.

Suggested First Steps

Curated data as publication and the corresponding data provenance and reliability could involve a major, long-term infrastructure project for the linguistic community. We would like to suggest a few simple first steps that the community as a whole can take. Our basic suggestion is to provide both carrots and sticks to the community and to pursue proactive education in data set publication. Although some linguists will become major contributors to curated data sets while others will play a more minor role, all linguists should understand and appreciate their importance: no linguist left behind. It will therefore be useful for the linguistic community to to engage in extensive dissemination and training efforts and to establish links with ongoing generic projects on metadata standards and preservation (e.g. PREMIS).

Encouraging students is not enough since the actions of successful researchers speak much more loudly than words. Therefore, if well-known, successful members of the linguistic community publish and share their data sets, this will set a powerful example for the next generation. Those who do publish their data should provide a simple "cite as" button with data to make it easy for those using the data to cite it in their works. This is particularly necessary since many linguists are unsure of the proper way in which to cite such data sets. Finally, it would be good to provide a service provision for data structure and integrity validation and for format conversion. Data structure and integrity validation can, at a minimum, check that the relevant metadata is in place before publication. As a more complex process, it can check over the data structure to be sure that it is correct and complete, e.g. a tab delineated file should have tabs between fields. A format conversion service would allow data set creators to publish their data sets in a variety of formats, both making it easier for researchers to use them and helping ensure the long term survival of the data as formats change.

There are also some institutional safeguards that can easily be put into place to ensure that data publication occurs and is done properly. First, publishers, editors, and reviewers can require provenance information for all data sources before agreeing to publish research work. Second, editors and funding agencies encourage data sets to be published. This is already starting but should be further encouraged. Also, the funding agencies should ensure that the data is in fact published if it was part of the grant and if possible should publicize that the data is now available. If grantees do not publish their data as promised, either the funding agency could withhold any future grants to the grantee and its institution or could not pay out the final part of the grant until the data is published, similar to how some dissertation fellowships reserve part of the grant until the signed dissertation is submitted.

Finally, the establishment of electronic data publishing journals in conjunction with a cyberinfrastructure should be considered, so as to provide a formal channel for establishing authorship of data sets and creating a scholarly reference in addition to a framework for peer review.

Events

Koenraad — Wed, 02 Sep 2009 03:37:37 CDT

Conferences and workshops related to cyberinfrastructure (chronological)

RILIVS workshop on infrastructure for linguistic variation
Oslo, 17-18 Sep. 2009
The workshop is part of a series of explorative workshops financed by NOS-HS under the heading "Research Infrastructure for Linguistic Variation Studies" (RILiVS). RILiVS is based on the networks Scandinavian Dialect Syntax (ScanDiaSyn), Nordic Centre of Excellence in Microcomparative Syntax (NORMS) plus the Medieval Nordic Text Archive (Menota), SweDia 2000, and a newly established network for the documentation of Sami languages led from the University of Tromsø.

Networking Event for European Research Infrastructures + Standards Workshop
Helsinki, 30 Sep. - 2 Oct. 2009
CLARIN would like to invite all interested European research infrastructure initiatives to exchange information to get a more comprehensive picture which will be essential for moving to a more coherent European eInfrastructure scenario.

2009 Annual Conference of the Alliance for Permanent Access
Den Haag, 24 Nov. 2009
The theme of the 2009 Annual Conference is: Keeping the Records of Science Accessible: towards a sustainable science data e-infrastructure!

e-Humanities workshop at 5th IEEE International Conference on e-Science
Oxford, 9-11 Dec. 2009
This e-Humanities track aims to showcase projects that contribute to e-Humanities, whether by providing integrated and interoperable infrastructures, or by offering new types of applications making use of such infrastructures and connecting the digital islands.

The seventh international conference on Language Resources and Evaluation (LREC 2010)
Malta, 17-23 May 2010 (incl. workshops)
The LREC 2010 Map of Language Resources, Technologies and Evaluation will be a collective enterprise of the LREC community, as a first step towards the creation of a very broad, community-built, Open Resource Infrastructure.

WG2: Resources

danmccloy — Mon, 31 Aug 2009 14:51:49 CDT

This page lists the resources referred to in the WG2 Wiki pages:

PLEASE CONTRIBUTE relevant links and resources

Encoding and Annotation Links:

Autotyp
ISO

My ISO Job: Guidance for Delegates and Experts (accessible document describing ISO and how experts can participate)

Joining In: Participating in International Standardization (another easy-to-read document on participating in ISO stnds. development)
Standards Development (webpage on standards development with links)

ISO Committees of interest to linguists:

TC 37/SC 4 - Language resource management
TC 37/SC 4 Working Groups
TC 37/SC 2 - Terminographical and lexicographical working methods
JTC 1/SC 2/WG2: Universal character set

ISO Script Codes (ISO 15924): http://www.unicode.org/iso15924/
OLAC
TypeCraft
Unicode

Unicode codecharts

http://www.unicode.org/charts/
Fonts
http://scripts.sil.org/IPAhome (Unicode-enabled fonts with IPA and other resources)
http://www.phon.ucl.ac.uk/home/wells/ipa-unicode.htm (John Wells' website on fonts with IPA and Unicode info)
http://linguistlist.org/sp/Fonts.html (LinguistList info on fonts)
Character pickers

http://people.w3.org/rishida/scripts/pickers/
http://weston.ruter.net/projects/ipa-chart/view/keyboard/
Keyboard and Input Methods
http://scripts.sil.org/UniIPAKeyboard
http://linguistlist.org/cfdocs/emeld/school/classroom/unicode/ipafont.htm (Tips on how to assign a keystroke for input and how to create a keyboard for character input)
http://www.phon.ucl.ac.uk/resource/phonetics/ (Unicode phonetic keyboard for PC with SIL fonts)
Recommendations on developing new orthographies

http://www.unicode.org/notes/tn19/
Unicode for Language Documentation (E-MELD document with useful information for linguists on Unicode)

XSL-transformations (EXtensible Stylesheet Language)

Data Collection Instruments Links:

Elicitation Materials Clearinghouse (Sociolinguistics elicitation instruments, U of Washington, Sociolinguistics Laboratory)
Typological tools for field linguists (questionnaires, etc., Max Planck Institute for Ev. Anthro., Leipzig):

http://www.eva.mpg.de/lingua/tools-at-lingboard/tools.php

Metadata Tagging Links:

Akustyk
Sociophonetics metadata tags

Web-Design Links:

RESTful design patterns (REpresentational State Transfer)

Storage Links:

SWORDS (based on the Atom Publishing Protocol)
Version control software:

Concurrent Versions System (CVS): An open-source revision control system
Subversion: An open-source revision control system
Microsoft Sharepoint Server, 2007

Infrastructure for Long-Term Archiving:
eSciDoc collaborative eResearch infrastructure
Fedora Commons Repository Software

Sharing Links:

WS/SOAP (standards for web services)

Access Links:

Wiki access control lists (for assignment of usage rights and privileges)

Bibliography:
DiPaolo, M. and Yaeger-Dror, M. (forthcoming) Best Practices in Sociophonetics. Cambridge UP

Existing Standards and Technologies

alexispalmer — Mon, 31 Aug 2009 13:25:35 CDT

See also pages for subfield-specific best practices
See also information re: participating in the development of ISO standards

Existing annotation standards and resources
This section lists the various annotation conventions and other resources for developing and discussing annotation standards that were suggested by participants of Cyberling09. It was copied from the working group 1 page on 23 July 2009, and ultimately (after the workshop whitepapers are published) this should be the permanent home for information on existing standards.

6.1. Phonetics and phonology

Phone segment tagging symbols, including both:

IPA and its various ASCII-fications, such as SAMPA, WorldBet

and language-specific phoneme-segment encodings such as ArpaBet for American English and the CSJ encoding for Japanese

the various ToBI conventions

6.2. Morphosyntax

Leipzig Glossing rules

ISO Morphosyntactic Annotation Format (MAF)
ISO Lexical Markup Framework (LMF): Homepage (with Publications and Tools)

Typecraft: a labeling system which, for any verb construction of a given language, provides a template for that construction type displaying its argument structure, in a fashion as transparent as possible. The template is constructed from a universally established inventory of labeling primitives.
http://www.typecraft.org/tc2wiki/Verbconstructions_cross-linguistically_-_Introduction

tags for short-unit word (SUW) and long-unit word (LUW) in the CSJ

6.3. Syntax and semantics

FrameNet annotation manual
FILL IN HERE FOR ENGLISH TREEBANK, etc., etc., etc.
ISO Syntactic Annotation Format (SynAF)
ISO Semantic Annotation Format - Time and Events (SemAF-TIME) (formerly TimeML)

6.4. Pragmatics and discourse structure

CHAT conventions for segmenting turns and identifying the participant and the setting

DAMSL
and other schemes documented and discussed at the 1998 DRI meeting

6.5. Gesture

David MacNeil's Gesture annotation
BTS sign transcription system
NEUROGES-ELAN system: workshop series homepage
[look at ANVIL page and LDC list for more?]

6.6. Other resources

ISO TC37 SC4 - Language Resource Management : Homepage

ISO Data Category Registry for linguistic concepts

GOLD (General Ontology for Linguistic Description)

various (lineages of) POS tagging systems such as:
text transcription symbols promoted on the LDC Corpus Cookbook page for transcription/symbols
International Standards for Language Engineering (ISLE)

ISO Linguistic Annotation Framework (LAF) Overview and Draft Standard (under revision)
XML Serialization for LAF : ISO Graph Annotation Format (GrAF)

XML Corpus Encoding Standard (XCES)

SIGAnn : ACL Special Interest Group on Annotations

NXT System: Switchboard in NXT NITE XML Toolkit Video lecture by Jean Carletta: the NITE XML Toolkit Meets the ICSI Meeting Corpus

tool and framework for building

E-MELD (Electronic Metastructure for Endangered Language Data): Homepage

All of the annotation systems listed on the COCOSDA Corpus Annotation Tools page

Existing standards for storage, retrieval, and search
This section was copied from the working group 2 page on 23 July 2009, and this information should ultimately live here.

STORAGE
-- text encoding standards

Unicode
Text Encoding Initiative (TEI) P5 Standard "These guidelines make recommendations about suitable ways of representing those features of textual resources which need to be identified explicitly in order to facilitate processing by computer programs. In particular, they specify a set of markers (or tags) which may be inserted in the electronic representation of the text, in order to mark the text structure and other features of interest."
...

(domain-specific) terminological standards

terminological standards (e.g., the GOLD ontology)
...

---- storage and retrieval standards

repository systems (e.g. www.escidoc.org and www.fedora-commons.org might be relevant in relation to long-term archiving
DELAMAN is 'as international umbrella body for archives and other initiatives with the goal of documenting and archiving endangered languages and cultures worldwide. Our aim is to stimulate interaction about practical matters that result from the experiences of fieldworkers and archivists, and to act as an information clearinghouse.'
The Rosetta Project is another archive actively promoting a set of best practices for storage of language data.

RETRIEVAL
-- reference/identification standards (i.e. metadata)

-- Citation Standards

COINS (a simple standard for embedding Dublin Core citation metadata in a web page)

SEARCH

Open Search. A very simple standard for sharing search results, usually by expressing such search results in the Atom Syndication Format. Although this is not the best standard (in terms of design, extensibility), it is relatively easy to adopt. Open Search also has proposed geographic extensions to describe how to query a collection based on geographic parameters.

ACCESS/REUSE

-- Cultural Heritage Global Schema and Ontologies

CIDOC (an ontology mainly applied by European museums and other heritage organizations that is nicely abstracted and very generalized, but is complex and has some difficulties in application)
OCHRE/ArchaeoML (a somewhat more simple global schema / ontology for cultural heritage applications, including archaeology, epigraphy and philology. It is highly abstract so that projects and collections retain native descriptive terminologies but some degree of interoperability and shared services are facilitated.

-- Copyright and Intellectual Property

Creative Commons provides a series of standard copyright licenses and associated metadata to explicitly give certain permissions and conditions for use/reuse of copyrighted content. These are useful to define how content can be used. However, these are complicated to apply with scientific data, since US copyright law makes a distinction between "facts" (ideas, concepts, objective data) and "expressions". Since many scientific datasets contain factual measurements and observation, they may not be protected by copyright. To make matters more complicated, the determination of what's a fact and what's an expression is ambiguous and a blurred distinction. This legal ambiguity and complexity makes it harder to use and reuse scientific data. Therefore, Creative Common's scientific arm, "Science Commons", recommends that scientists do not use Creative Commons copyright licenses for scientific data. Instead, Science Commons recommends that scientific application explicitly dedicate data to the public domain using the "CC-Zero" declaration. CC-Zero removes legal ambiguity around data, removes all restrictions for reuse, and in theory, maximizes the scientific value of data.

-- APIs/standards for interfaces with other resources (e.g. corpora, lexica/lexical resources, treebanks?, ...)

WordNet "This document presents a standard conversion of Princeton WordNet to RDF/OWL. It describes how it was converted and gives examples of how it may be queried for use in Semantic Web applications."

Existing tools, web services, and other technologies
This section was copied in part from the working group 3 page on 23 July 2009.

TOOLS

TypeCraft Collaborative text annotation
WALS The World Atlas of Language Structures Online
ODIN Online Database of INterlinear glossed text

TextGrid "TextGrid aims to create a community grid for the collaborative editing, annotation, analysis and publication of specialist texts. It thus forms a cornerstone in the emerging e-Humanities."

Natural Language Toolkit (NLTK) "Open source Python modules, linguistic data and documentation for research and development in natural language processing, supporting dozens of NLP tasks, with distributions for Windows, Mac OSX and Linux."
eHumanities Desktop (project is in alpha development stage, no description available yet)
Roma: TEI validation tool "These pages will help you design your own TEI validator, as a DTD, RELAXNG or W3C Schema."
Chorus is a version control system designed to enable workflows appropriate for typical language development teams who are geographically distributed. Chorus is a Palaso Project.
e-Linguistics: building a cyberinfrastructure for linguistics (including a Python toolkit for data migration; documentation is still being posted)
Consistent Document Engineering Toolkit
Thai language specific tools

SWATH Thai word segmentation and POS tagging
CU Thai word segmentation
CU Thai Romanization
iSpeech Automatic speech recognition toolkit
Vaja Thai/English text-to-speech synthesis engine
LIBS String kernel based language-script-encoding (LSE) identifier for 85 LSEs
KUI Collaborative WordNet translation platform
Web service and Open API for Asian WordNet, Multilingual Morphological Analyzer, and Thai Soundex

General purpose tools in use by linguists:

R Project for statistical computing and the linguistics packages in EMU.
Praat doing phonetics by computer
Python
ANVIL video annotation research tool

WEB SERVICES
-- Representational state transfer (REST)

A design pattern used as a standard in which clients and servers are able to communicate over the internet.
Communication is limited to the four verbs of the HTTP Protocol: GET, POST, PUT, DELETE. These four commands are used in manipulating any resource on the internet that has a URI. The limitation of only four commands provides simplicity in the semantics of communication between the client and server.

Resource: Any entity; anything that can be identified with a URI (e.g. phone number, car, person, idea).
GET: Retrieve one or many resources from the server.
POST: Update a resource on the server.
PUT: Add a new resource to the server.
DELETE: Destroy a resource from the server.
Note that GET, POST, PUT and DELETE are passed in the header as methods of an HTTP request. GET is the default method when submitting a URL request using a web browser.

MultiTree Examples:

GET http://multitree.linguistlist.org/codes/pol

Semantics: “Get the code resource ‘pol’ from the multitree.linguistlist.org server.”
Returns an HTML formatted page of all data pertaining to the ‘pol’ code resource in the MultiTree repository

GET http://multitree.linguistlist.org/codes/pol/trees.json

Semantics: “Get all tree resources that contain the code resource ‘pol’ and return in JSON format.
Returns a javascript object that lists all tree resources which contain the code ‘pol’.

JSON: Javascript object notation used for marking up data. Similar to XML but far more succinct since its purpose is for communicating between machines with disregard for human readability.

It is possible to implement a RESTful web service from the ground up, though several popular frameworks help facilitate its construction:

Ruby on Rails (ruby)
Django (python)
ZEND (php)
Restlet (java)

REST can be compared to other standards of communication over the internet: RPC and SOAP. The critique of these standards, though which have been used more widely than REST, is that they add unnecessary complexity when designing both communication interfaces between clients and servers.
See also: Web Standards

OTHER TECHNOLOGIES
-- here

Getting Involved in ISO Standards Development

alexispalmer — Mon, 31 Aug 2009 13:23:46 CDT

The topic of standards has been raised in several Working Groups, and a number ISO standards have been mentioned at Cyberling 2009 (e.g., MAF, ISO 639 [language tags], and ISOcat [ISO 12620]). The discussion of standards raises a more general issue, not separately addressed: how can linguists participate in ISO standards development and why is it important they do?

What is ISO and why should linguists get Involved in ISO standards development?

The International Organization for Standardization (http://www.iso.org) is a very large organization with standards covering everything from screw thread specifications to language tags.

Because standards development is “based on voluntary involvement of all interests in the market place,” it is very important for linguists to participate in the creation of any standard that relates to language. The involvement of linguists can ensure that the ISO standards have been reviewed carefully and meet linguists’ needs. For some ISO standards, such as character encoding, it is also crucial that user communities be directly consulted. This is particularly true for users of less well-known languages (and their writing systems). Since linguists often work with such user communities, they can act as intermediaries to encourage such user community participation. For groups of linguists that have been developing their own set of standards independent of ISO, funneling such work into an ISO standard will help guarantee long-term stability and longevity.

The ISO standards process

The process of developing and approving a standard can be quite lengthy, lasting several years. Hence participation requires time, commitment, and dedication from those who wish to participate.

Meetings can be held in different locations throughout the world, so funding for travel is also a desideratum, though attending meeting is not required. Attendance is useful, however, as it is an opportunity to observe first-hand how ISO Working Groups function. It also enables participants to meet face-to-face, voice their concerns, and discuss topics in depth, which can be more difficult if handled via email. Since standard development is done through consensus, talking with others in-person can help move the process of coming to agreement more quickly.

How to participate in ISO standards development

Read up on the ISO standards development process (such as http://www.iso.org/iso/my_iso_job.pdf) and other docs on the ISO website (such as http://www.iso.org/iso/joining_in_2007.pdf and http://www.iso.org/iso/standards_development.htm )

Contact the Convener of the Working Group (WG) you are interested in to see if you can participate (see list below). Read the relevant standard the WG is working on and related documents.
Contact your country’s standards organization (http://www.iso.org/iso/iso_members), describe your expertise, and see if you can participate in the Working Group in an official capacity.

Attend the Working Group meetings, if at all possible. The Working Group Convener can permit “invited guests” to participate in meetings. There may be a limitation on the number of times you can participate as an “invited guest,” depending upon the rules of your standards organization, so if you feel you can commit to long-term participation, try to get appointed as a representative or an “expert” by your member body.

Participating in the Unicode Standard's development

The description above focuses on how linguists can participate in ISO standards development. For linguists interested in character encoding, Unicode is another way to participate in standards development, as Unicode and ISO 10646 are completely synchronized. All characters accepted into ISO 10646 must also be approved by the Unicode Technical Committee (UTC).

For some linguists, participation in Unicode might be more accessible. UTC meetings are held quarterly, often in the San Francisco Bay Area. In contrast, the ISO Working Group that oversees work on ISO 10646 (JTC 1/SC 2/WG2) meets twice a year, often in different locations throughout the world.

The UTC discussions can be quite technical in nature, as UTC members – drawn largely from the computer industry – need to be sure that UTC decisions can be implemented on current platforms and in software, and are in line with Unicode policies. Still, input from linguists and members of the user community on character and script proposals is valuable and highly encouraged; feedback is particularly welcome through the Unicode website form.

Ways to get involved:

To pose a question on Unicode or to keep an eye on Unicode discussions, sign up for the public Unicode email list
To become more involved in Unicode development, become a member of Unicode, either as a representative of your institution, as an individual, or as a student. This will give you access to Unicode documents and the "inner" Unicode email list (Unicore).
If you are available, attend a Unicode Technical Committee meeting as an observer (or, as a member, see above). Contact Deborah Anderson, UC Berkeley representative to the Unicode Consortium, for further information (dwanders at berkeley dot edu [with no spaces]).
Submit feedback on the Unicode Standard, its other specifications, or any aspect of the Unicode website via the Unicode website form.

References: ISO Technical committees (TC) / Subcommittees (SC) / Joint Technical Committees (JTC) of Interest to Linguists

TC 37/SC 4 - Language resource management (work on feature structures, morpho-syntactic annotation framework, linguistic annotation framework, word segmentation of written texts, syntactic annotation framework, semantic annotation framework)

Working Group 1: Basic descriptors and mechanisms for language resources

Working Group 2: Representation Schemes

Working Group 3: Multilingual text representation

Working Group 4: Lexical database

Working Group 5: Workflow of language resource management

TC 37/SC 2 - Terminographical and lexicographical working methods – codes for the representation of names of languages (ISO 639), terminological entries in standards, lexicographical production and marketing, interpreting/interpretation processes,

Working Group 1: Language coding

Working Group 2: Terminography

Working Group 3: Lexicography

Working Group 4: Source identification for language resources

Working Group 5: Requirements and certification schemes for cultural diversity management

Working Group 6: Translation and Interpretation Services

JTC1/SC2 Coded character sets

Working Group 2: Universal coded character set

Genealogical classification with AutoTyp

alexispalmer — Mon, 31 Aug 2009 07:54:13 CDT

Case study subdiscipline:	Typology
Project title:	Autotyp typological databases
Software used:	FileMaker Pro (TM)
Goals of this case study:	Demonstrate the use of notes and a log to track changes in the genealogical classification in the database.

See slideshow: jn_cyberling_reanalysis_trail.ppt (attachment to this page)

(The genealogical classification can also be accessed online at http://uni-leipzig.de/~autotyp )

This is a series of screenshots of records in the Autotyp typological database. One of the modules there is a genealogy for every language we have in the database (about 2600 languages so far). By now the genealogy amounts to a near-complete list of all the world's language families and most of their major subgrouping. It is under constant revision as fieldwork and comparative work discover new languages and families, join families into larger groups, and otherwise change classification. The literature is also filled with speculative proposals about genealogical relationships that do not have adequate support. Keeping track of the revisions is not particularly burdensome and is absolutely essential for documenting the grounds for the classification. (Unlike most other classifications of the world's languages, ours is based on evidence and criteria for non-chance degrees of resemblance, so every decision does need to be described.)

We have a Notes field on all of our genealogical classification records, and this is used to record the history of editing and reclassifying. The slides in this slideshow show:

3. The levels of classification we consider determinate. Stock and language entries are required for every record.

4. The search interface as seen for a language search from the on-line search interface.

6. Sample language record for Arabic, with its classification at different levels. Note that Afroasiatic, though a bona fide family, is not a stock but a higher-level grouping (demonstrable but not reconstructable).

8-15 (see commentary slides 7 and 14): These are a set of screenshots of records for languages of the Penutian macrogroup. They document our changing decisions on the status of this group and the subgrouping of some of its components.

16-17 Page from the database log outlining the changes made to Penutian in 2008.

17-18 Slide showing evidence and counterevidence presented in Notes comments, on the subclassification of Western Malayo-Polynesian.

WG2: Subfield-specific practices

alexispalmer — Mon, 31 Aug 2009 07:23:33 CDT

Standards are great, now how can I use them?

Widespread use of standards and/or best practices just won't happen unless it is easy for people to:

Locate information re: standards and what they entail.
Learn how to apply the standards to their own data.

The page(s) linked below provide common practices in some specific subfields of Linguistics:

Sociolinguistics
Typology
Language documentation and description

Phonetics (article stub)
Syntax (article stub)

------------------------------------------------------------------------------------------------------------

PLEASE CONTRIBUTE!
This is an opportunity to share your expertise regarding subfield-specific standards or best practices, how these can be applied in your field, and why one should bother doing so. We are seeking contributions to this page and its offspring, in two forms:
1. Expansion of existing seed lists
2. Creation of new seed lists

WG2: Case Studies

alexispalmer — Mon, 31 Aug 2009 07:20:13 CDT

The case studies developed by Working Group 2 are intended to serve as illustrations of the following topics as these have arisen and been addressed in specific subfields of linguistics:

Unicode character encoding standards for increasing stable display, readability, and sharing of data
Relational database storage
Wiki-based sharing of research
Metadata tags for increased transparency and usability of data
Version control
Web standards for sharing datasets
Machine reusability of language data

Case Studies:

PLEASE CONTRIBUTE!
We welcome the contribution of additional case studies regarding standards for data storage, search, and retrieval (and other relevant topics... there are many).

Reasons for Using Unicode

DeborahAnderson — Sat, 29 Aug 2009 14:05:43 CDT

Why use Unicode?

Linguistic data should be created and stored using standards. For written text, the character encoding standard is Unicode (http://unicode.org) /ISO 10646.

Linguists who create text using a non-Unicode font run the risk of jeopardizing their data, for it won't be easily found by widely used search processes, nor will it be saved in a stable, standardized format that will guarantee longevity.

PDFs, Word documents, webpages, and other documents that are put online but are created using a non-Unicode font will cause problems when doing searching. Using images for missing letters or symbols also present a problem, as in the following example.

The snippets below come from the article “A Preliminary Study of Jaw Movement in Arrernte Consonant Production” by Marija Tabaina (Journal of the International Phonetic Association 2009, 39: 33-51). There are two versions available: an HTML version and a PDF.

The HTML version uses images for certain IPA symbols, including one for the voiceless retroflex stop “ʈ”.

Unfortunately, it is not possible to search on the images in the HTML document itself, or by doing a “Google” search across the Internet on such an image, so one won't be able to locate f “ʈ” in the HTML document.

In the PDF version of the same text, a non-Unicode font has been used for the “ʈ”. At first glance, the “ʈ” appears fine in the PDF:

However, the font used has put the glyph for “ʈ“(Unicode LATIN SMALL LETTER T WITH RETROFLEX HOOK, U+0288) in the spot that is properly allocated in Unicode for the DAGGER (†, U+2020). As a result, it is not possible to search in this document for the “ʈ” (Unicode U+0288), because the “ʈ” has been overlaid on the dagger character. (In a similar way, many old Greek fonts would put the lowercase alpha on top of Latin "a" in the font, lowercase beta on top of "b", etc. To search for alpha, one had to search on the Latin letter "a". With Unicode, alpha has its own number [codepoint], which is different from the Latin lowercase "a", so now it is possible to search for alpha separately from Latin "a".)

A non-Unicode font will present other problems for the user: If one copies and pastes the letter “ʈ” from the above PDF into a Unicode-compliant word-processing document, it appears as a dagger.

The above example demonstrates that using a non-Unicode font can prevent search engines from finding documents and text properly.

Another important factor is that old documents created with non-Unicode fonts will be hard to read in the future, since they won’t be based on an international standard. If an elderly linguist were to die and leave his data on his computer (which he had keyed in with his own non-standard font), it may take considerable time and effort to convert the data into a standardized format that is usable by others, with the possibility that the data could be lost forever.

Tools (Fonts/Keyboards/etc.):

The listing below is not comprehensive, but is only intended to provide a few reliable webpages that provide tools and other useful information on Unicode-enabled products.

Fonts

Most core fonts that come with recent operating systems are Unicode-based, but they may not include all the special characters required by linguists. (Note: A new font that will be released with Windows 7, Ebrima, will include improved support for many African languages that use the Latin script. It also include the Vai and N'Ko scripts.)

Unicode-enabled fonts with IPA from SIL: http://scripts.sil.org/IPAhome (Note: SIL fonts that are prefixed with "SIL IPA" are non-Unicode fonts)
John Well's webpage with a listing of IPA fonts (and Unicode input info): http://www.phon.ucl.ac.uk/home/wells/ipa-unicode.htm
LinguistList info on fonts: http://linguistlist.org/sp/Fonts.html

"Character pickers"

These enable users to select the letters or symbols they wish, and cut and paste them into documents

http://people.w3.org/rishida/scripts/pickers/

Keyboards and Inputting Methods:

Keyboard info from SIL: http://scripts.sil.org/UniIPAKeyboard
Keyboards and Inputting Tips from E-MELD: http://linguistlist.org/cfdocs/emeld/school/classroom/unicode/ipafont.htm
IPA Keyboards for the PC from the Speech, Hearing, and Phonetic Sciences Dept., University College London: http://www.phon.ucl.ac.uk/resource/phonetics/

Other Script Standards and Useful Resources:

ISO Script Codes (ISO 15924): http://www.unicode.org/iso15924/

Recommendations on the Development of New Orthographies: http://www.unicode.org/notes/tn19/
This is a set of guidelines for linguists who are devising orthographies, so the orthography can be accessible to users on computers

Unicode for Language Documentation
An informative page from E-MELD with sections on (a) adding characters to Unicode, (b) precomposed forms (letters made up of a base character + one or more diacritic, and why these are not in Unicode), and (c) IPA and Unicode.

How to Get Involved in ISO Standards Development
This document also includes a short section on how to get involved in the development of the Unicode Standard

WG2: Big ideas

DeborahAnderson — Sat, 29 Aug 2009 13:10:04 CDT

On this page:How do we define 'other standards'?
An aside...Standards vs. Best Practices
How do we encourage adoption of standards in linguistics?

Data sharing: the publication model
Standards are great, now how do I use them?
How can I participate in the creation of ISO standards?

How do we define 'other standards'?

For the purposes of this workshop, we take our domain of interest to be standards related to the sharing of language data within the linguistics community. The discussion is organized around a questionable* division of data sharing into four subtopics:

*We call this a questionable division because the four are deeply interrelated.

Storage of digital data
Retrieval and discoverability of digital data (i.e. discoverability at the document or resource level)
Search of digital data (i.e. discoverability at the within-document level)
Access and reusability of digital data

The table below displays some of the key areas of concern for each of the four topics we take to fall within our domain of interest.

STORAGE

RETRIEVAL

SEARCH

ACCESS/REUSE

METADATA

DIGITIZATION of both primary data & metadata

VERSIONING tracking major changes/decisions made and motivations for same

SOURCE MATERIALS linking to audio/video source

CITATION STANDARDS how to cite datasets

FORMATS & STANDARDS open access standards & formats

DIGITAL FINGERPRINTING

** ANNOTATION CONVENTIONS collection and dissemination of conventions used in existing data collections

PRIVACY/LEGAL ISSUES related to user access, privilege assignment, copyright and data ownership

LEGACY DATA providing within-subfield model for best practices data sharing

ADAPTIVE CODING ability to adjust data coding scheme as knowledge evolves

CONSISTENCY consistency and quality of data annotations

** SUBFIELD-SPECIFIC USABILITY CONCERNS specialized standards, metadata sets, ontologies, etc.

PUBLICATION

STABLE ADDRESSING OF RESOURCES

REUSABILITY repurposing of data for use in addressing new research questions by both humans and machine

ARCHIVING

WEB STANDARDS

** indicates action items listed in the table that may be implemented immediately or in the near term.

An aside...standards vs. best practices (a distinction with a difference?)

We weren’t certain that linguistics, as a field or academic culture, has a tradition of clearly differentiating standards from best practices. We do not have an organizational body within the field that sets standards. Subfields are autonomous and vary to the extent that standards or best practices are discussed, named and adhered to. We nonetheless considered a few possible distinctions that might generally be made between the two, so we can be as clear as possible about what we mean when we use the terms “standard” and “best practice” in these wiki pages:

STANDARDS:

Often, these are theory-neutral conventional systems for accomplishing some task (often related to analysis, description, or publication) in linguistics (e.g., the IPA system for phonetic transcription)
Named (so practitioners may name standards to which their practices adhere in published work, for example)
Official (new standards will explicitly obsoletize prior or existing ones) handed down from a high-level organization charged with regulating usage, nomenclature, etc.
Use is subject to sanction or mandate
Developed over time via a process involving the deliberations of an organizational body of experts, after discussion and consensus
Follow from best practices, ranked and subjected to selection
links: discussions regarding standards
links: political issues

BEST PRACTICES:

Often, principled practices rather than mandated systems for accomplishing some analytical, descriptive, or publication-related task in linguistics
Recommended, but not strictly enforced
Generated by practitioners in a bottom-up process, who wish to build consensus in practice and are often interested in motivating the need for a particular practice

How do we encourage adoption of standards in linguistics?

Data sharing: the publication model offers some possibilities with regard to building incentives for adopting standards, acknowledging use of annotated corpora, receiving and giving credit for the use of annotated and marked-up data (as a scholarly practice of value to the field). Working group 5 explored ways that other disciplines are sharing data, so we may learn from these examples.
Publication mechanisms for linguistic data collections are one possibility for encouraging adoption of standards.

Receiving academic credit for publication of data would provide a needed incentive for doing the extra work needed to be sure that standards are followed.
Peer review will improve the quality of shared data.
Publication and proper citation of data facilitate demonstrating the scholarly contribution made by providing the data.
Publication of legacy data would provide a valuable training ground for young researchers as well as providing a model for preparation of data according to best practices.

Standards are great, now how can I use them?
Widespread use of standards and/or best practices just won't happen unless it is easy for people to:

Locate information re: standards and what they entail.
Learn how to apply the standards to their own data.

Of course, a commitment to communication, collaboration, coordination, community building and open access to data are crucial for supporting the use of standards. Working Group 7 discussed this issue.

How can I participate in the creation of ISO standards?
ISO is home to a wide array of standards, and the process of standardization can appear to be opaque and daunting to the outsider. A short page devoted to how linguists can participate in ISO standards development is located at: How to Get Involved in ISO Standards Development. (This page also includes a short section on how to participate in the development of the Unicode Standard.)

By actively participating in ISO standardization, linguists will have a vested interest in the use of such standards. Involvement by linguists also has the result of making sure standards are suited to current needs, and haven't become fossilized. Ideally, participants should get recognition from their host institution for work on standards development, a job that often requires many hours of time and (at times) considerable personal expense.

Future work:
WG2: Big ideas -- not yet written up

simple but powerful tools
privacy and ethics concerns must be considered

Typology common practices

Johanna.Nichols — Fri, 28 Aug 2009 23:22:19 CDT

Common practices:
Several institutions and projects have large databases with cross-linguistic typological information on a number of languages. (List with links to be added.) The information has generally been gathered over some years from published grammars, one's own fieldwork, consultations with experts, etc. and represents an extremely large outlay of time and resources. In some cases language data is solicited in return for authorship credit for that portion of the database. Ways that database owners might provide peer review for these contributions have occasionally been discussed but probably never implemented.
Several projects make some or all of their data available, via a search interface or as downloadable data files. There are no field-wide standards for content of fields, data format, etc. but this does not seem to be an obstacle to finding and using data. It is standard to link data to language names using ISO codes. There is a growing consensus that data is most useful if it is free of one's own lumpings (thus, e.g., not "large/medium/small" for inventories of elements such as morphological cases or phonemes, but the actual number; not "VO/OV" but an actual specific basic word order).

Needed practices:
Consensus on how to give database authors and owners credit for their work while also making data publicly available. Practices include: Make data publicly available and request a citation. Offer data on request in exchange for citation. Offer data on request in exchange for coauthorship. (All three models are used in other fields.)
Peer review for databases as a whole. (This is separate from the question of how the database owner obtains peer review for individual contributions to the database.) Accuracy of entries and usefulness, comprehensiveness, empirical and theoretical adequacy, etc. of data categories need review.
Appropriate credit (on one's CV and with one's employing institution or prospective employer) for creation and maintenance of databases.

Language documentation common practices

Johanna.Nichols — Fri, 28 Aug 2009 19:50:40 CDT

NOTE: this structure is offered only as a suggested skeleton. Please modify as you see fit.

Introduction: The fieldworker (and this includes documentary linguists) needs to learn the structure of the language and produce a grammatical description, dictionary, and texts usable by researchers and others (including interested members of the speech community). He or she is usually also singlehandedly responsible for producing theoretical and comparative work on the language. Not much will be said here about producing grammars (though see Dryer 2006) or the general scholarly work, other than to note that the whole set of responsibilities gives the fieldworker a considerably heavier workload than the average for linguists, so tools and standards that save time are especially needed. The tasks in need of standards and tools for sharing and access are: recording and/or digitizing; transcribing; creating texts; creating a lexicon; and archival storage and access of recordings, texts, and lexicon (and perhaps other materials).

Recommended Readings:
Dryer, Matthew S. 2006. Descriptive theories, expanatory theories, and basic linguistic theory. In Felix Ameka, Alan Dench and Nicholas Evans, eds., Catching Language: Issues in Grammar Writing. Berlin: Mouton de Gruyter.

Software:
(See Existing standards... below.)

Existing standards, common practices, and/or best practices:
Recording: Information available from the DoBeS project of the Max Planck Institute for Psycholinguistics, Nijmegen; the Hans Rausing Endangered Languages Project, SOAS, London; and others.

Transcribing: Many field linguists use Transcriber, available from SourceForge.net.

Text work. The main tasks are inputting or importing transcribed material, morphological interlinearizing, syntactic annotation, parsing, lemmatization, concordance building, and preparation for corpus searching and archiving. For standards for some kinds of interlinearization and annotation see the report of Working Group 1. To my knowledge there are no current widely used software tools that could be described as best practice, though there is a growing consensus about what they need to do (e.g. multiple annotation tiers; link to recordings; enable non-interlinear annotation for discontinuous, non-compositional, multi-word, and non-linear categories and functions; enable printout of text segments with standard publishable interlinears; enable corpus searches of all kinds; enable other access). There is also very little knowledge of what might go into theory-neutral syntactic annotation.
A list of tools dated 2004 and including some text tools is here. I am aware of these updates and additions: Kura; and a glossing tool under development by Thomas Mayer (presented at ALT8, July 2009). The DoBeS project has elaborate and fairly specialized tools for text and dictionary work.

Dictionary compilation. The most common practice for compiling dictionaries of all kinds (descriptive, defining, etymological, etc.) seems to be use of commercial database software to create a self-standing database that does not, e.g., link to a text corpus.

Archival storage and access: Archives have their own standards for metadata and data formats. The DoBeS project has an extensive list of archives with links.

Needed standards and data-sharing resources:

Easy-to-use tools for text work are badly needed. Basic research, leading ultimately to standards, for theory-neutral syntactic annotation is needed.