CL-ReLKT (Crosslingual-Retrieval Language Knowledge Transfer)

CL-ReLKT: Cross-lingual Language Knowledge Transfer for Multilingual Retrieval Question Answering, NAACL-2022 (Finding)

Motivation

Cross-Lingual Retrieval Question Answering (CL-ReQA) is concerned with retrieving answer documents or passages to a question written in a different language. A common approach to CL-ReQA is to create a multilingual sentence embedding space such that question-answer pairs across different languages are close to each other.

In this paper, our goal is to improve the robustness of multilingual sentence embedding that works with a wide range of languages, including those with a limited amount of training data. Leveraging the generalizability of knowledge distillation, we propose a Cross-Lingual Retrieval Language Knowledge Transfer (CL-ReLKT) framework.

Paper

Link: https://openreview.net/forum?id=y42xxJ_xx8 (Not the final version)

Citation

@inproceedings{limkonchotiwat-etal-2022-cl-relkt,
    title = "{CL-ReKD}: Cross-lingual Language Knowledge Transfer for Multilingual Retrieval Question Answering",
    author = "Limkonchotiwat, Peerat  and
      Ponwitayarat, Wuttikorn  and
      Udomcharoenchaikit, Can  and
      Chuangsuwanich, Ekapol  and
      Nutanong, Sarana",
    booktitle = "Findings of the North American Chapter of the Association for Computational Linguistics: NAACL 2022"
    month = jun,
    year = "2022",
    publisher = "Association for Computational Linguistics",
}

Model and Datasets

• Model: CLICK HERE
• Datasets: CLICK HERE
• Docker: Coming Soon

How to train

Step1: Triplet loss warmup step

• Run warmup.sh
• In this step, we finetune the mUSE model with our training data (i.e., XORQA, MLQA, or XQUAD) where the anchor is the question, the positive is the answer of question, and the negative is obtained from bm25.

Step2: Triplet loss online training

• Run teacher.sh
• In this step, we continute finetune the model in Step1 by using the concept of online mining to obtain the negative samples.

Step3: Language Knowledge Transfer (Distillation)

• Run distillation.sh
• In this step, we initilize the model's weight from Step2 and finetuning it with language knowledge transfer technique (Section2.2).
• We use 3 terms minimization such as question(English)-question(Non-English), document-document, document-question(non-English) as shown in the figure:

Where all settings of each step are posted in the script file.

Performance