Automatic construction of a Kiswahili corpus from the World Wide Web

Abstract

A corpus is a large collection of language data either in written form or spoken form or both. It can be used to construct a language model that is used in many language technology applications. Some of these include speech to text, optical character recognition, machine translation and spell checking. The easiest way to create a text corpus is by putting together electronic text documents. For most languages, getting a huge collection of electronic texts is a time-consuming and challenging task. The monotonous nature of such a task will inevitably lead to much less attention being paid to the errors that might find their way into the text collection. This paper describes the working of an application that was used to build a Kiswahili corpus from the Internet to be used in natural language processing applications.