Publications

• Energy-based Automated Model Evaluation
  Ru Peng*, Heming Zou*, Haobo Wang, Yawen Zeng, Zenan Huang, Junbo Zhao.
  International Conference on Learning Representations (ICLR), 2024.
  Abstract PDF Code BibTeX
     The conventional evaluation protocols on machine learning models rely heavily on a labeled, i.i.d-assumed testing dataset, which is not often present in real-world applications. The Automated Model Evaluation (AutoEval) shows an alternative to this traditional workflow, by forming a proximal prediction pipeline of the testing performance without the presence of ground-truth labels. Despite its recent successes, the AutoEval frameworks still suffer from an overconfidence issue, substantial storage and computational cost. In that regard, we propose a novel measure --- Meta-Distribution Energy (MDE) that allows the AutoEval framework to be both more efficient and effective. The core of the MDE is to establish a meta-distribution statistic, on the information (energy) associated with individual samples, then offer a smoother representation enabled by energy-based learning. We further provide our theoretical insights by connecting the MDE with the classification loss. We provide extensive experiments across modalities, datasets and different architectural backbones to validate MDE's validity, together with its superiority compared with prior approaches. We also prove MDE's versatility by showing its seamless integration with large-scale models, and easy adaption to learning scenarios with noisy- or imbalanced- labels.
  @article{peng2024energy,
      title={Energy-based Automated Model Evaluation},
      author={Peng, Ru and Zou, Heming and Wang, Haobo and Zeng, Yawen and Huang, Zenan and Zhao, Junbo},
      journal={arXiv preprint arXiv:2401.12689},
      year={2024}
  }
  
  
  
• CAME: Contrastive Automated Model Evaluation
  Ru Peng*, Qiuyang Duan*, Haobo Wang, Jiachen Ma, Yanbo Jiang, Yongjun Tu, Xiu Jiang, Junbo Zhao.
  IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  Abstract PDF Code BibTeX
     The Automated Model Evaluation (AutoEval) framework entertains the possibility of evaluating a trained machine learning model without resorting to a labeled testing set. Despite the promise and some decent results, the existing AutoEval methods heavily rely on computing distribution shifts between the unlabelled testing set and the training set. We believe this reliance on the training set becomes another obstacle in shipping this technology to real-world ML development. In this work, we propose Contrastive Automatic Model Evaluation (CAME), a novel AutoEval framework that is rid of involving training set in the loop. The core idea of CAME bases on a theoretical analysis which bonds the model performance with a contrastive loss. Further, with extensive empirical validation, we manage to set up a predictable relationship between the two, simply by deducing on the unlabeled/unseen testing set. The resulting framework CAME establishes a new SOTA results for AutoEval by surpassing prior work significantly.
  @inproceedings{peng2023came,
      title={Came: Contrastive automated model evaluation},
      author={Peng, Ru and Duan, Qiuyang and Wang, Haobo and Ma, Jiachen and Jiang, Yanbo and Tu, Yongjun and Jiang, Xiu and Zhao, Junbo},
      booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
      pages={20121--20132},
      year={2023}
  }
  
  
  
• Oral Paper, top 4.1%
  Distill The Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation
  Ru Peng*, Yawen Zeng*, Junbo Zhao.
  Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
  Abstract PDF Code BibTeX
     Past works on multimodal machine translation (MMT) elevate bilingual setup by incorporating additional aligned vision information. However, an image-must requirement of the multimodal dataset largely hinders MMT’s development — namely that it demands an aligned form of [image, source text, target text].This limitation is generally troublesome during the inference phase especially when the aligned image is not provided as in the normal NMT setup. Thus, in this work, we introduce IKD-MMT, a novel MMT framework to support the image-free inference phase via an inversion knowledge distillation scheme. In particular, a multimodal feature generator is executed with a knowledge distillation module, which directly generates the multimodal feature from (only) source texts as the input. While there have been a few prior works entertaining the possibility to support image-free inference for machine translation, their performances have yet to rival the image-must translation. In our experiments, we identify our method as the first image-free approach to comprehensively rival or even surpass (almost) all image-must frameworks, and achieved the state-of-the-art result on the often-used Multi30k benchmark. Our code and data are availableat: https://github.com/pengr/IKD-mmt/tree/master.
  @inproceedings{peng2022distill,
      title={Distill The Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation},
      author={Peng, Ru and Zeng, Yawen and Zhao, Jake},
      booktitle={Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
      pages={2379--2390},
      year={2022}
  }
  
  
  
• Oral Paper
  HybridVocab: Towards Multi-Modal Machine Translation via Multi-Aspect Alignment
  Ru Peng*, Yawen Zeng*, Junbo Zhao.
  International Conference on Multimedia Retrieval (ICMR), 2022.
  Abstract PDF Code BibTeX
     Multi-modal machine translation (MMT) aims to augment the linguistic machine translation frameworks by incorporating aligned vision information. As the core research challenge for MMT, how to fuse the image information and further align it with the bilingual data remains critical. Existing works have either focused on a methodological alignment in the space of bilingual text or emphasized the combination of the one-sided text and given image. In this work, we entertain the possibility of a triplet alignment, among the source and target text together with the image instance. In particular, we propose Multi-aspect AlignmenT (MAT) model that augments the MMT tasks to three sub-tasks --- namely cross-language translation alignment, cross-modal captioning alignment and multi-modal hybrid alignment tasks. Core to this model consists of a hybrid vocabulary which compiles the visually depictable entity (nouns) occurrence on both sides of the text as well as the detected object labels appearing in the images. Through this sub-task, we postulate that MAT manages to further align the modalities by casting three instances into a shared domain, as compared against previously proposed methods. Extensive experiments and analyses demonstrate the superiority of our approaches, which achieve several state-of-the-art results on two benchmark datasets of the MMT task.
  @inproceedings{peng2022hybridvocab,
      title={HybridVocab: Towards Multi-Modal Machine Translation via Multi-Aspect Alignment},
      author={Peng, Ru and Zeng, Yawen and Zhao, Junbo},
      booktitle={Proceedings of the 2022 International Conference on Multimedia Retrieval},
      pages={380--388},
      year={2022}
  }
  
  
  
• Oral Paper
  Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network
  Ru Peng, Nankai Lin, Yi Fang, Shengyi Jiang, Tianyong Hao, Boyu Chen, Junbo Zhao.
  International Conference on Neural Information Processing (ICONIP), 2022.
  Abstract PDF Code BibTeX
     Syntax knowledge contributes its powerful strength in Neural machine translation (NMT) tasks. Early NMT works supposed that syntax details can be automatically learned from numerous texts via attention networks. However, succeeding researches pointed out that limited by the uncontrolled nature of attention computation, the NMT model requires an external syntax to capture the deep syntactic aware- ness. Although existing syntax-aware NMT methods have born great fruits in combining syntax, the additional workloads they introduced ren- der the model heavy and slow. Particularly, these efforts scarcely involve the Transformer-based NMT and modify its core self-attention network (SAN). To this end, we propose a parameter-free, Dependency-scaled Self-Attention Network (Deps-SAN) for syntax-aware Transformer-based NMT. A quantified matrix of dependency closeness between tokens is constructed to impose explicit syntactic constraints into the SAN for learning syntactic details and dispelling the dispersion of attention distributions. Two knowledge sparsing techniques are further integrated to avoid the model overfitting the dependency noises introduced by the ex- ternal parser. Experiments and analyses on IWSLT14 German-to-English and WMT16 German-to-English benchmark NMT tasks verify the effectiveness of our approach.
  @inproceedings{peng2022deps,
      title={Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network},
      author={Peng, Ru and Lin, Nankai and Fang, Yi and Jiang, Shengyi and Hao, Tianyong and Chen, Boyu and Zhao, Junbo},
      booktitle={International Conference on Neural Information Processing},
      pages={26--37},
      year={2022},
      organization={Springer}
  }
  
  
  
• Syntax-Aware Attentional Neural Machine Translation Directed by Syntactic Dependency Degree
  Ru Peng, Tianyong Hao, Yi Fang.
  Neural Computing and Applications (NCA), 2020.
  Abstract PDF BibTeX
     There are various ways to incorporate syntax knowledge into neural machine translation (NMT). However, quantifying the dependency syntactic intimacy (DSI) between word pairs in a dependency tree has not being considered to use in attentional and transformer-based NMT. In this paper, we innovatively propose a variant of Tree-LSTM to capture the syntactic dependency degree (SDD) between word pairs in dependency trees. Two syntax-aware distances, including a tuned syntax distance and a p-dependent distance, are proposed. For attentional NMT, two syntax-aware attentions based on two syntax-aware distances are proposed for attentional NMT, and we also design a dual attention to simultaneously generate global context and dependency syntactic context. For transformer-based NMT, we explicitly incorporate the dependency syntax into self-attention network (SAN) to propose a syntax-aware SAN. Experiments on IWSLT’17 English–German, IWSLT Chinese–English and WMT’15 English–Finnish translation tasks show that our syntax-aware NMT significantly improves translation quality by comparing with baseline methods, even the state-of-the-art transformer-based NMT.
  @article{peng2021syntax,
      title={Syntax-aware neural machine translation directed by syntactic dependency degree},
      author={Peng, Ru and Hao, Tianyong and Fang, Yi},
      journal={Neural Computing and Applications},
      volume={33},
      number={23},
      pages={16609--16625},
      year={2021},
      publisher={Springer}
  }
  
  
  
• Best Paper Candidates
  Neural Machine Translation with Attention Based on a New Syntactic Branch Distance
  Ru Peng, Zhitao Chen, Tianyong Hao, Yi Fang.
  China Conference on Machine Translation (CCMT), 2019.
  Abstract PDF BibTeX
     Attention mechanism has been proved to be able to improve the quality of neural machine translation by selectively focusing on partial words of a source sentence during translation process. Attention mechanism usually focuses on local attention by using solely the linear index distance of words while ignores syntax structures of sentences. In this paper, we extend local attention through syntax distance constraint, and propose an attention mechanism based on a new syntactic branch distance, which simultaneously pays attention to words with similar linear index distances and syntax-related words. Based on the English-to-German translation task, experiment results showed that our model outperforms a recent baseline method with an improvement of 1.61 BLEU points, demonstrating the effectiveness of the proposed model.
  @inproceedings{peng2019neural,
      title={Neural machine translation with attention based on a new syntactic branch distance},
      author={Peng, Ru and Chen, Zhitao and Hao, Tianyong and Fang, Yi},
      booktitle={Machine Translation: 15th China Conference, CCMT 2019, Nanchang, China, September 27--29, 2019, Revised Selected Papers 15},
      pages={47--57},
      year={2019},
      organization={Springer}
  }