<link rel="stylesheet" href="styles.f3b1fba60ec7970c.css">

Publication:
Smoothing a tera-word language model

Loading...
Thumbnail Image

Departments

Item type:Organizational Unit,

School / College / Institute

Item type:Organizational Unit,

Program

KU-Authors

Organization Authors

Co-Authors

Date

Language

Embargo Status

No

Journal Title

Journal ISSN

Volume Title

Alternative Title

Abstract

Frequency counts from very large corpora, such as the Web 1T dataset, have recently become available for language modeling. Omission of low frequency n-gram counts is a practical necessity for datasets of this size. Naive implementations of standard smoothing methods do not realize the full potential of such large datasets with missing counts. In this paper I present a new smoothing algorithm that combines the Dirichlet prior form of (Mackay and Peto, 1995) with the modified back-off estimates of (Kneser and Ney, 1995) that leads to a 31% perplexity reduction on the Brown corpus compared to a baseline implementation of Kneser-Ney discounting.

Source

Publisher

Association for Computational Linguistics

Citation

item.page.haspartof

Source

ACL-08: HLT - 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

item.page.ispartofseries

item.page.edition

DOI

item.page.datauri

item.page.link

Rights

Other

Copyrights Note

Rights and licensing

Other

Endorsement

Review

Supplemented By

Referenced By

Related Patent

Related Goal

Google Scholar
Scholar'da Ara ↗
Bu yayında DOI yok — Altmetric/Dimensions/PlumX/BIP! rozetleri DOI gerektirir.