Human Perception in Natural Language Generation

Lorenzo De Mattei; Huiyuan Lai; Felice Dell’Orletta; Malvina Nissim

doi:10.18653/v1/2021.gem-1.2

Human Perception in Natural Language Generation

Lorenzo De Mattei, Huiyuan Lai, Felice Dell’Orletta, Malvina Nissim

Correct Metadata for

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, and observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language.

Anthology ID:: 2021.gem-1.2
Volume:: Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
Month:: August
Year:: 2021
Address:: Online
Editors:: Antoine Bosselut, Esin Durmus, Varun Prashant Gangal, Sebastian Gehrmann, Yacine Jernite, Laura Perez-Beltrachini, Samira Shaikh, Wei Xu
Venue:: GEM
SIG:: SIGGEN
Publisher:: Association for Computational Linguistics
Note:
Pages:: 15–23
Language:
URL:: http://aclanthology.org/2021.gem-1.2/
DOI:: 10.18653/v1/2021.gem-1.2
Bibkey:
Cite (ACL):: Lorenzo De Mattei, Huiyuan Lai, Felice Dell’Orletta, and Malvina Nissim. 2021. Human Perception in Natural Language Generation. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 15–23, Online. Association for Computational Linguistics.
Cite (Informal):: Human Perception in Natural Language Generation (De Mattei et al., GEM 2021)
Copy Citation:
PDF:: http://aclanthology.org/2021.gem-1.2.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{de-mattei-etal-2021-human,
    title = "Human Perception in Natural Language Generation",
    author = "De Mattei, Lorenzo  and
      Lai, Huiyuan  and
      Dell{'}Orletta, Felice  and
      Nissim, Malvina",
    editor = "Bosselut, Antoine  and
      Durmus, Esin  and
      Gangal, Varun Prashant  and
      Gehrmann, Sebastian  and
      Jernite, Yacine  and
      Perez-Beltrachini, Laura  and
      Shaikh, Samira  and
      Xu, Wei",
    booktitle = "Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "http://aclanthology.org/2021.gem-1.2/",
    doi = "10.18653/v1/2021.gem-1.2",
    pages = "15--23",
    abstract = "We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, and observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="de-mattei-etal-2021-human">
    <titleInfo>
        <title>Human Perception in Natural Language Generation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Lorenzo</namePart>
        <namePart type="family">De Mattei</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Huiyuan</namePart>
        <namePart type="family">Lai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Felice</namePart>
        <namePart type="family">Dell’Orletta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Malvina</namePart>
        <namePart type="family">Nissim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Antoine</namePart>
            <namePart type="family">Bosselut</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Esin</namePart>
            <namePart type="family">Durmus</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Varun</namePart>
            <namePart type="given">Prashant</namePart>
            <namePart type="family">Gangal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sebastian</namePart>
            <namePart type="family">Gehrmann</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yacine</namePart>
            <namePart type="family">Jernite</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Laura</namePart>
            <namePart type="family">Perez-Beltrachini</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Samira</namePart>
            <namePart type="family">Shaikh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Wei</namePart>
            <namePart type="family">Xu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, and observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language.</abstract>
    <identifier type="citekey">de-mattei-etal-2021-human</identifier>
    <identifier type="doi">10.18653/v1/2021.gem-1.2</identifier>
    <location>
        <url>http://aclanthology.org/2021.gem-1.2/</url>
    </location>
    <part>
        <date>2021-08</date>
        <extent unit="page">
            <start>15</start>
            <end>23</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T Human Perception in Natural Language Generation
%A De Mattei, Lorenzo
%A Lai, Huiyuan
%A Dell’Orletta, Felice
%A Nissim, Malvina
%Y Bosselut, Antoine
%Y Durmus, Esin
%Y Gangal, Varun Prashant
%Y Gehrmann, Sebastian
%Y Jernite, Yacine
%Y Perez-Beltrachini, Laura
%Y Shaikh, Samira
%Y Xu, Wei
%S Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
%D 2021
%8 August
%I Association for Computational Linguistics
%C Online
%F de-mattei-etal-2021-human
%X We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, and observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language.
%R 10.18653/v1/2021.gem-1.2
%U http://aclanthology.org/2021.gem-1.2/
%U http://doi.org/10.18653/v1/2021.gem-1.2
%P 15-23

Download as File

Markdown (Informal)

[Human Perception in Natural Language Generation](http://aclanthology.org/2021.gem-1.2/) (De Mattei et al., GEM 2021)

Human Perception in Natural Language Generation (De Mattei et al., GEM 2021)

ACL

Lorenzo De Mattei, Huiyuan Lai, Felice Dell’Orletta, and Malvina Nissim. 2021. Human Perception in Natural Language Generation. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 15–23, Online. Association for Computational Linguistics.