logseq/pages/hls__ase2022-paper108_1661352547766_0.md

file-path:: ../assets/ase2022-paper108_1661352547766_0.pdf
file:: [ase2022-paper108_1661352547766_0.pdf](../assets/ase2022-paper108_1661352547766_0.pdf)
title:: hls__ase2022-paper108_1661352547766_0

- n-gram-base
  ls-type:: annotation
  hl-page:: 1
  id:: 63063a7c-1f1f-4833-9a37-20f185180632
- allows for precisely and efficiently measuring the similarity of code
  ls-type:: annotation
  hl-page:: 1
  id:: 63063aa0-134b-43ac-a304-0de166e31190
- language-agnosti
  ls-type:: annotation
  hl-page:: 1
  id:: 63063aae-e6f7-492b-ac1b-7c685e8771eb
- handle incomplete or partially incorrect code
  ls-type:: annotation
  hl-page:: 1
  id:: 63063ab3-3683-42b4-ab50-23dd74f3704e
- efficient
  ls-type:: annotation
  hl-page:: 1
  id:: 63063ab5-108f-4eaf-acd0-1b7334e6016e
- automated program repai
  ls-type:: annotation
  hl-page:: 1
  id:: 63063cad-79f4-45da-8134-64ac39b638da
- predicting code from natural language descriptions of the desired functionality
  ls-type:: annotation
  hl-page:: 1
  id:: 63063cb3-dc58-49e2-b55c-1fa1234f333c
- bilingual evaluation understudy
  ls-type:: annotation
  hl-page:: 1
  id:: 63063cd3-3c09-45f8-bf15-444de176fc9c
- e find at least 21 papers published since 2015 that use BLEU as a metric to evaluate code prediction
  ls-type:: annotation
  hl-page:: 1
  id:: 63063d0f-c404-455d-91eb-9f7481e1db50
- The basic idea of BLEU is to compare a prediction against one or more reference solutions by computing the overlap between n-grams, i.e., contiguous sequences of code tokens
  ls-type:: annotation
  hl-page:: 1
  id:: 63063d27-0931-46bb-89ba-6436d127cdbf
- trivially shared n-grams, i.e., n-grams that occur across code written in the same language without implying any deeper relationship or semantic similarity. 
  ls-type:: annotation
  hl-page:: 1
  id:: 63063df8-a6dc-4bee-96b2-2512719da00b
- This paper presents CrystalBLEU, a metric to precisely and efficiently evaluate code similarity despite trivially shared n-grams.
  ls-type:: annotation
  hl-page:: 1
  id:: 63063e31-2f9f-4a2b-9308-d98c5c9b71d3
- The approach is an extension of BLEU that removes trivially shared n-grams before computing the n-gram overlap between two pieces of code
  ls-type:: annotation
  hl-page:: 1
  id:: 63063e49-ec06-4fb1-8a5f-567100bf353e
- To identify trivially shared n-grams, CrystalBLEU analyzes a corpus of code and identifies n-grams that occur frequently across many examples
  ls-type:: annotation
  hl-page:: 1
  id:: 630642a3-0ec8-4822-8ae9-1c7c7f9ab432
- distinguishability
  ls-type:: annotation
  hl-page:: 1
  id:: 630642b3-af01-4920-8ea7-a8e4fe046264
- distinguishability measures how much more similar code examples known to be semantically equivalent are compared to code examples that are not equivalent to each other.
  ls-type:: annotation
  hl-page:: 2
  id:: 630642cc-fe7c-48e7-8633-ef1149f8f512
- Concode
  ls-type:: annotation
  hl-page:: 2
  id:: 630642eb-1e39-4764-83c2-f36c4d42c4a1
- BigCloneBench
  ls-type:: annotation
  hl-page:: 2
  id:: 630642ee-9b86-4ec4-866e-d0143aa8d660
- it is more effective at distinguishing semantically similar code from code merely written in the same language.
  ls-type:: annotation
  hl-page:: 2
  id:: 63064327-6edb-4a1d-9e22-12f553aa941f
- most common n-grams in programming languages appear relatively more often than the most common n-grams in natural languages, and that they appear in many different programs regardless of their semantic similarity.
  ls-type:: annotation
  hl-page:: 2
  id:: 63064c06-55b1-45a7-bda0-b7c861303d12
- distinguishability
  ls-type:: annotation
  hl-page:: 2
  id:: 63064c0a-1933-4f6f-ae9b-04d65cafd1dd
- CrystalBLEU, which ignores trivially shared n-grams when comparing two pieces of code
  ls-type:: annotation
  hl-page:: 2
  id:: 63064c19-dc64-4ece-b112-ddfbe89fc638
- modified precision
  ls-type:: annotation
  hl-page:: 2
  id:: 63064e2b-f532-437f-a1a6-6466f735d592
- odified precision
  ls-type:: annotation
  hl-page:: 3
  id:: 6309dae9-1d7b-43c2-8f7b-2a5312d1b9c1
- trivially shared n-grams,
  ls-type:: annotation
  hl-page:: 4
  id:: 6309dafd-1db8-44d6-9e31-aef476b63017
- 325
  ls-type:: annotation
  hl-page:: 5
  id:: 6309dcb8-0ad1-4050-bc82-47e116305357
- We now present the CrystalBLEU metric, which in contrast to BLEU, increases distinguishability by accounting for trivially shared ngrams
  ls-type:: annotation
  hl-page:: 5
  id:: 6309e00a-1ea8-4ea1-aa59-94490799d9ec
- we identify trivially shared n-grams. 
  ls-type:: annotation
  hl-page:: 5
  id:: 6309e00f-9a1a-46dd-954c-0a7eb0e906bf
- BLEU score that accounts for the trivially shared n-grams
  ls-type:: annotation
  hl-page:: 5
  id:: 6309e012-ff0e-45bb-acee-870461906a76
- How well does CrystalBLEU distinguish between similar and dissimilar code?
  ls-type:: annotation
  hl-page:: 6
  id:: 6309e040-49ed-4a07-ab50-74b7174ff025
- avoid misleading results provided by BLEU?
  ls-type:: annotation
  hl-page:: 6
  id:: 6309e043-4d6e-41d4-abb2-6ff76202290b
- ShareCode: Semantically equivalent, human-written code.
  ls-type:: annotation
  hl-page:: 6
  id:: 6309e049-b50a-4077-8946-7ef78d2d86f7
- ShareCode
  ls-type:: annotation
  hl-page:: 7
  id:: 6309e069-4d1b-4350-b0fe-648bcd92100d
- BigCloneBench
  ls-type:: annotation
  hl-page:: 7
  id:: 6309e07b-2a59-4dd1-930a-779695afda54
- Concode: Code generation task.
  ls-type:: annotation
  hl-page:: 7
  id:: 6309e085-79eb-4784-8423-63ad6e96d128
- CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code
  ls-type:: annotation
  hl-page:: 1
  id:: 6309e970-d42e-4be1-9735-9c239019a46a