Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Update Scoring for Speech Summarization from NLG-Eval to Huggingface Evaluate#5341

Merged
mergify[bot] merged 6 commits intoespnet:masterfrom
roshansh-cmu:scorefix
Jul 22, 2023
Merged

Update Scoring for Speech Summarization from NLG-Eval to Huggingface Evaluate#5341
mergify[bot] merged 6 commits intoespnet:masterfrom
roshansh-cmu:scorefix

Conversation

@roshansh-cmu
Copy link
Contributor

@roshansh-cmu roshansh-cmu commented Jul 22, 2023

The NLG-Eval doesn't seem to be updated recently and is incompatible with the latest versions of python, numpy.

This PR modifies all scoring to rely on the huggingface evaluate package instead.

Note:

  • The ROUGE-L scores are slightly higher than previously obtained using NLG-Eval. However, the NLP community uses evaluate nowadays to compute scores, so it makes sense to retain this.
  • I also updated local/score.sh because the ls command format does not work with certain shell types
  • I removed any checks on nlgeval and bert_score wherever they appear and added the evaluate package to setup.py for installation.

@roshansh-cmu
Copy link
Contributor Author

roshansh-cmu commented Jul 22, 2023

Requesting a review from @Jungjee if you have any time ? Thanks !

Jungjee
Jungjee reviewed Jul 22, 2023
View reviewed changes
Copy link
Contributor

@Jungjee Jungjee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just added one minor comment.

setup.py
"fairscale",
"transformers",
"gtn==0.0.0",
"evaluate",
Copy link
Contributor

@Jungjee Jungjee Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add the version that worked for you?

Copy link
Contributor Author

@roshansh-cmu roshansh-cmu Jul 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the evaluate codebase is updated with transformers, so maybe getting the latest version like transformers is best ?

@codecov
Copy link

codecov bot commented Jul 22, 2023 •
edited
Loading

Codecov Report

Merging #5341 (14acebf) into master (8ed5d5c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #5341   +/-   ##
=======================================
  Coverage   76.11%   76.11%           
=======================================
  Files         672      672           
  Lines       59859    59859           
=======================================
  Hits        45561    45561           
  Misses      14298    14298
Flag Coverage Ξ”
test_integration_espnet1 65.93% <ΓΈ> (ΓΈ)
test_integration_espnet2 47.92% <ΓΈ> (ΓΈ)
test_python 66.40% <ΓΈ> (ΓΈ)
test_utils 23.17% <ΓΈ> (ΓΈ)

Flags with carried forward coverage won't be shown. Click here to find out more.

πŸ“£ We’re building smart automated test selection to slash your CI/CD build times. Learn more

@sw005320 sw005320 added this to the v.202307 milestone Jul 22, 2023
@sw005320
Copy link
Contributor

sw005320 commented Jul 22, 2023

LGTM!

@sw005320 sw005320 added auto-merge Enable auto-merge SSUM Speech summarization labels Jul 22, 2023
@sw005320
Copy link
Contributor

sw005320 commented Jul 22, 2023

I also made a label SSUM by considering the growth of this area. This is the first PR/issue having it.

@mergify mergify bot merged commit 79a74bc into espnet:master Jul 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@Jungjee Jungjee Jungjee left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

auto-merge Enable auto-merge ESPnet2 Installation Need review SSUM Speech summarization

Projects

None yet

Milestone

v.202307

Development

Successfully merging this pull request may close these issues.

3 participants

@roshansh-cmu @sw005320 @Jungjee