Welcome to the SemEval-2023 Task-1 - Visual Word Sense Disambiguation (Visual-WSD)

Join the mailing group: vwsd@googlegroups.com

Citation:

@inproceedings{raganato-etal-2023-semeval,
    title = "{S}em{E}val-2023 {T}ask 1: {V}isual {W}ord {S}ense {D}isambiguation",
    author = "Raganato, Alessandro  and
      Calixto, Iacer and
      Ushio, Asahi and
      Camacho-Collados, Jose  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
}

Overview of some SemEval Participants systems provided by the participants themselves, including a short description and GitHub link (whether available)

Baseline organizers - GitHub link

Codalab competition page

Ranking Final (Test) Phase of the Visual-WSD Competition 2023 (Updated Feb 6, 2023)

[TRAIN+TRIAL] Download train+trial data - Google drive link - OneDrive link [17GB] (Updated Oct 18, 2022): Train and trial data in English language, including gold keys.

[TRIAL] Download trial data - Google drive link (Updated Sept 06, 2022): Trial data only (no train) for the English language, including gold keys.

[TEST] Download test images - Google drive link - OneDrive link [10.4GB] (Updated Jan 14, 2023): Test images in their original size.

[TEST] Download test data - Google drive link (Updated Feb 08, 2023): Test data (queries) for the English, Farsi and Italian languages, including gold keys.

[TEST] Download test images resized - Google drive link - OneDrive link [572MB] (Updated Jan 14, 2023): These are the same images in the test set, but resized to a smaller size. Participants may choose to use either one or the other for their final submission(s).

License: The dataset is released under the CC-BY-NC 4.0 license.

Fair data usage policy: We require users participating in our shared task to adhere to a fair data usage policy. All users agree that they will not attempt to search the trial/training/test data using any search engine on the web, to reverse engineer the data generation process, or to tamper with the data beyond the goals of the task.

Task: Given a word and some limited textual context, the task is to select among a set of candidate images the one which corresponds to the intended meaning of the target word.

Example: Given the full phrase andromeda tree containing the ambiguous target word andromeda, and the following ten candidate images, the task is to select the corresponding one. In this case, the correct image is the first one on the left, as shown below.

Gold image:

Organizers of the shared task: