Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Submitter and non-submitter definition for unverified result disclosures #157

@psyhtest

Description

@psyhtest

The MLPerf Results Messaging Guidelines contain the following:

Submitters (i.e., those who have submitted results for review and verification by MLCommons) are not allowed to publish any results for a given benchmark version before its official publication date.

For example, if an organization only submits ResNet50 results to the upcoming v3.1 round, they are not allowed to disclose these results until the publication date. (However, they can seemingly obtain a similar set of results for a previous version of the benchmark e.g. v3.0, and disclose them as unverified results anytime they like, including during the v3.1 review period.)

Non-submitters (i.e., those who have not submitted results for review and verification by MLCommons) are not allowed to publish any results until two weeks after the official publication date for that benchmark version.

Now, if the same organization , who has only submitted ResNet50 results, obtains some unverified results, say, for GPT3, can they disclose them right after the official publication date, without waiting for two weeks?

On the one hand, the answer could be yes: this organization is a submitter, since they have submitted something to this round, just not this benchmark. On the other hand, the answer should be no: they have submitted nothing for this benchmark.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions