【Huggingface Transformers】英語の要約タスクの推論と評価方法

from google.colab import drive 
drive.mount('/content/drive')
!mkdir -p '/content/drive/My Drive/huggingface_transformers_demo/'
%cd '/content/drive/My Drive/huggingface_transformers_demo/'

!git clone https://github.com/huggingface/transformers
%cd transformers

!pip install git+https://github.com/huggingface/transformers

# Huggingface Datasetsのインストール
!pip install datasets

!pip install sentencepiece

!pip install sacrebleu

#「The CNN/DailyMail Dataset」をダウンロード
from datasets import load_dataset

dataset = load_dataset("cnn_dailymail", version="3.0.0")

# サンプルデータを取り出す
sample = dataset["train"][1]

# 記事の長さと本文を出力
print(len(sample["article"]),sample["article"][:500])

# 要約文の長さと本文を出力
print(len(sample["highlights"]),sample["highlights"])

4051 Editor's note: In our Behind the Scenes series, CNN correspondents share their experiences in covering news and analyze the stories behind the events. Here, Soledad O'Brien takes users inside a jail where many of the inmates are mentally ill. An inmate housed on the "forgotten floor," where many mentally ill inmates are housed in Miami before trial. MIAMI, Florida (CNN) -- The ninth floor of the Miami-Dade pretrial detention facility is dubbed the "forgotten floor." Here, inmates with the most s
281 Mentally ill inmates in Miami are housed on the "forgotten floor" Judge Steven Leifman says most are there as a result of "avoidable felonies"While CNN tours facility, patient shouts: "I am the son of the president"Leifman says the system is unjust and heghting for change .

sample_text = dataset["train"][1]["article"][:2000]
# We'll collect the generated summaries of each model in a dictionary
summaries = {}

import nltk
from nltk.tokenize import sent_tokenize
nltk.download("punkt")

from transformers import pipeline, set_seed
set_seed(42)

#　元のデータセットの要約（答え）を表示
print(dataset["train"][1]["highlights"])

Mentally ill inmates in Miami are housed on the "forgotten floor"
Judge Steven Leifman says most are there as a result of "avoidable felonies"
While CNN tours facility, patient shouts: "I am the son of the president"
Leifman says the system is unjust and he's fighting for change .

# GPT-2による要約
pipe = pipeline("text-generation", model="gpt2")
gpt2_query = sample_text + "\nTL;DR:\n"
pipe_out = pipe(gpt2_query, max_length=512, clean_up_tokenization_spaces=True)
summaries_gpt2 =  "\n".join(sent_tokenize(pipe_out[0]["generated_text"][len(gpt2_query) :]))
print(summaries_gpt2)

-1.
The 9th degree refers to a prisoner who is "forgotten" because of his illness
-2.
As this sentence makes no sense, it probably doesn't hold much of a weight.
Most inmates on the 9th cannot handle an indefinite term of imprisonment, and only about 20 to 30% are charged with a mental illness.
Many are on death's

# T5による要約
pipe = pipeline("summarization", model="t5-large")
pipe_out = pipe(sample_text)
summaries_t5 = "\n".join(sent_tokenize(pipe_out[0]["summary_text"]))
print(summaries_t5)

mentally ill inmates are housed on the ninth floor of a florida jail .
most face drug charges or charges of assaulting an officer .
judge says arrests often result from confrontations with police .
one-third of all people in Miami-dade county jails are mental ill .

# 　BARTによる要約
pipe = pipeline("summarization", model="facebook/bart-large-cnn")
pipe_out = pipe(sample_text)
summaries_bart = "\n".join(sent_tokenize(pipe_out[0]["summary_text"]))
print(summaries_bart)

Mentally ill inmates are housed on the "forgotten floor" of Miami-Dade jail.
Most often, they face drug charges or charges of assaulting an officer.
Judge Steven Leifman says the arrests often result from confrontations with police.
He says about one-third of all people in the county jails are mentally ill.

# PEGASUSによる要約
pipe = pipeline("summarization", model="google/pegasus-cnn_dailymail")
pipe_out = pipe(sample_text)
summaries_pegasus = pipe_out[0]["summary_text"].replace(" .<n>", ".\n")
print(summaries_pegasus)

Mentally ill inmates in Miami are housed on the "forgotten floor"<n>The ninth floor is where they're held until they're ready to appear in court.
Most often, they face drug charges or charges of assaulting an officer.
They end up on the ninth floor severely mentally disturbed .

!pip install rouge_score

from datasets import load_metric
rouge_metric = load_metric("rouge")

records1 = []
rouge_names = ["rouge1", "rouge2", "rougeL", "rougeLsum"]

rouge_metric.add(prediction=summaries_gpt2, reference=dataset["train"][1]["highlights"])
score = rouge_metric.compute()
rouge_dict = dict((rn, score[rn].mid.fmeasure) for rn in rouge_names)
records1.append(rouge_dict)

print(records1)

[{'rouge1': 0.27522935779816515, 'rouge2': 0.01869158878504673, 'rougeL': 0.14678899082568808, 'rougeLsum': 0.27522935779816515}]

rouge_metric.add(prediction=summaries_t5, reference=dataset["train"][1]["highlights"])
score = rouge_metric.compute()
rouge_dict = dict((rn, score[rn].mid.fmeasure) for rn in rouge_names)
records2.append(rouge_dict)

print(records2)

[{'rouge1': 0.3829787234042553, 'rouge2': 0.13043478260869565, 'rougeL': 0.2553191489361702, 'rougeLsum': 0.3829787234042553}, {'rouge1': 0.3829787234042553, 'rouge2': 0.13043478260869565, 'rougeL': 0.2553191489361702, 'rougeLsum': 0.3829787234042553}, {'rouge1': 0.3829787234042553, 'rouge2': 0.13043478260869565, 'rougeL': 0.2553191489361702, 'rougeLsum': 0.3829787234042553}]

rouge_metric.add(prediction=summaries_bart, reference=dataset["train"][1]["highlights"])
score = rouge_metric.compute()
rouge_dict = dict((rn, score[rn].mid.fmeasure) for rn in rouge_names)
records3.append(rouge_dict)

print(records3)

[{'rouge1': 0.4752475247524753, 'rouge2': 0.22222222222222224, 'rougeL': 0.31683168316831684, 'rougeLsum': 0.4158415841584159}, {'rouge1': 0.4752475247524753, 'rouge2': 0.22222222222222224, 'rougeL': 0.31683168316831684, 'rougeLsum': 0.4158415841584159}, {'rouge1': 0.4752475247524753, 'rouge2': 0.22222222222222224, 'rougeL': 0.31683168316831684, 'rougeLsum': 0.4158415841584159}]

rouge_metric.add(prediction=summaries_pegasus, reference=dataset["train"][1]["highlights"])
score = rouge_metric.compute()
rouge_dict = dict((rn, score[rn].mid.fmeasure) for rn in rouge_names)
records4.append(rouge_dict)

print(records4)

[{'rouge1': 0.3232323232323232, 'rouge2': 0.2061855670103093, 'rougeL': 0.2828282828282828, 'rougeLsum': 0.3232323232323232}, {'rouge1': 0.3232323232323232, 'rouge2': 0.2061855670103093, 'rougeL': 0.2828282828282828, 'rougeLsum': 0.3232323232323232}, {'rouge1': 0.3232323232323232, 'rouge2': 0.2061855670103093, 'rougeL': 0.2828282828282828, 'rougeLsum': 0.3232323232323232}]