IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2312.14565.html
   My bibliography  Save this paper

Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy

Author

Listed:
  • Johann Laux
  • Fabian Stephany
  • Alice Liefgreen

Abstract

The global surge in AI applications is transforming industries, leading to displacement and complementation of existing jobs, while also giving rise to new employment opportunities. Data annotation, encompassing the labelling of images or annotating of texts by human workers, crucially influences the quality of a dataset directly influences the quality of AI models trained on it. This paper delves into the economics of data annotation, with a specific focus on the impact of task instruction design (that is, the choice between rules and standards as theorised in law and economics) and monetary incentives on data quality and costs. An experimental study involving 307 data annotators examines six groups with varying task instructions (norms) and monetary incentives. Results reveal that annotators provided with clear rules exhibit higher accuracy rates, outperforming those with vague standards by 14%. Similarly, annotators receiving an additional monetary incentive perform significantly better, with the highest accuracy rate recorded in the group working with both clear rules and incentives (87.5% accuracy). In addition, our results show that rules are perceived as being more helpful by annotators than standards and reduce annotators' difficulty in annotating images. These empirical findings underscore the double benefit of rule-based instructions on both data quality and worker wellbeing. Our research design allows us to reveal that, in our study, rules are more cost-efficient in increasing accuracy than monetary incentives. The paper contributes experimental insights to discussions on the economical, ethical, and legal considerations of AI technologies. Addressing policymakers and practitioners, we emphasise the need for a balanced approach in optimising data annotation processes for efficient and ethical AI development and usage.

Suggested Citation

  • Johann Laux & Fabian Stephany & Alice Liefgreen, 2023. "Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy," Papers 2312.14565, arXiv.org, revised Aug 2024.
    • Handle: RePEc:arx:papers:2312.14565
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2312.14565
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Kaplow, Louis & Shavell, Steven, 2002. "Economic analysis of law," Handbook of Public Economics, in: A. J. Auerbach & M. Feldstein (ed.), Handbook of Public Economics, edition 1, volume 3, chapter 25, pages 1661-1784, Elsevier.
    2. Isaac Ehrlich & Richard A. Posner, 1974. "An Economic Analysis of Legal Rulemaking," The Journal of Legal Studies, University of Chicago Press, vol. 3(1), pages 257-286, January.
    3. Stephany, Fabian & Teutloff, Ole, 2024. "What is the price of a skill? The value of complementarity," Research Policy, Elsevier, vol. 53(1).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giuseppe Dari-Mattiacci & Bruno Deffains, 2007. "Uncertainty of Law and the Legal Process," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 163(4), pages 627-656, December.
    2. Magnus Söderberg, 2008. "Uncertainty and regulatory outcome in the Swedish electricity distribution sector," European Journal of Law and Economics, Springer, vol. 25(1), pages 79-94, February.
    3. Christian Gollier & James Hammitt & Nicolas Treich, 2013. "Risk and choice: A research saga," Journal of Risk and Uncertainty, Springer, vol. 47(2), pages 129-145, October.
    4. Mondello, Gérard, 2012. "La responsabilité environnementale des prêteurs : difficultés juridiques et ensemble des possibles," L'Actualité Economique, Société Canadienne de Science Economique, vol. 88(2), pages 257-278, Juin.
    5. Eduard Hartwich & Alexander Rieger & Johannes Sedlmeir & Dominik Jurek & Gilbert Fridgen, 2023. "Machine economies," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-13, December.
    6. Brennan, Timothy & Boyd, James, 1996. "Pluralism and Regulatory Failure: When Should Takings Trigger Compensation?," RFF Working Paper Series dp-96-09, Resources for the Future.
    7. Dietrich Earnhart & Sarah Jacobson & Yusuke Kuwayama & Richard T. Woodward, 2023. "Discretionary Exemptions from Environmental Regulation: Flexibility for Good or for Ill," Land Economics, University of Wisconsin Press, vol. 99(2), pages 203-221.
    8. Louis Kaplow, 1992. "A Model of the Optimal Complexity of Rules," NBER Working Papers 3958, National Bureau of Economic Research, Inc.
    9. Gérard Mondello & Evens Salies, 2016. "Tort law under oligopolistic competition," SciencePo Working papers Main hal-03459225, HAL.
    10. Ogus, Anthony, 2001. "Regulatory Institutions and Structures," Centre on Regulation and Competition (CRC) Working papers 30704, University of Manchester, Institute for Development Policy and Management (IDPM).
    11. Francesco Parisi, 2004. "Positive, Normative and Functional Schools in Law and Economics," European Journal of Law and Economics, Springer, vol. 18(3), pages 259-272, December.
    12. Oliver Hart & John Moore, 2004. "Agreeing Now to Agree Later: Contracts that Rule Out but do not Rule In," Edinburgh School of Economics Discussion Paper Series 109, Edinburgh School of Economics, University of Edinburgh.
    13. Draca, Mirko & Nathan, Max & Nguyen-Tien, Viet & Oliveira-Cunha, Juliana & Rosso, Anna & Valero, Anna, 2024. "The New Wave? The Role of Human Capital and STEM Skills in Technology Adoption in the UK," The Warwick Economics Research Paper Series (TWERPS) 1521, University of Warwick, Department of Economics.
    14. Leonardo Felli & Alessandro Riboni & Luca Anderlini, 2007. "Statute Law or Case Law?," 2007 Meeting Papers 952, Society for Economic Dynamics.
    15. Gérard Mondello, 2012. "The Equivalence of Strict Liability and Negligence Rule: A " Trompe l'œil " Perspective," Post-Print hal-00727223, HAL.
    16. Lin, Ming-Jen, 2009. "More police, less crime: Evidence from US state data," International Review of Law and Economics, Elsevier, vol. 29(2), pages 73-80, June.
    17. Müller, Daniel & Schmitz, Patrick W., 2015. "Overdeterrence of repeat offenders when penalties for first-time offenders are restricted," Economics Letters, Elsevier, vol. 129(C), pages 116-120.
    18. Horn, Henrik & Mavroidis, Petros C & Nordström, Håkan, 1999. "Is The Use Of The WTO Dispute Settlement System Biased?," CEPR Discussion Papers 2340, C.E.P.R. Discussion Papers.
    19. Bruno Deffains & Marie Obidzinski, 2009. "Real Options Theory for Law Makers," Recherches économiques de Louvain, De Boeck Université, vol. 75(1), pages 93-117.
    20. Simon Johnson & Andrei Shleifer, 1999. "Coase v. the Coasians," Harvard Institute of Economic Research Working Papers 1885, Harvard - Institute of Economic Research.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2312.14565. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.
    pFad - Phonifier reborn

    Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

    Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


    Alternative Proxies:

    Alternative Proxy

    pFad Proxy

    pFad v3 Proxy

    pFad v4 Proxy