Skip to main content | Skip to Navigation | Text Size : | Language:

logo of Linguistic Data Consortium for Indian Languages (LDC-IL)
Corpora and Technological Resources for Urdu | Official Website of Linguistic Data Consortium for Indian Languages

Corpora and Technological Resources for Urdu

Maintained by : Shahnawaz Alam
Last Updated on: 17/01/2025


Introduction

Urdu is one of the prominent language used in the Indian sub-continent. It belongs to the Indo-Aryan family. Urdu is influenced by Arabic and Persian. Urdu is written in the Perso-Arabic script. On the other hand region-wise Urdu language is co-existed side by side mostly in the northern part of India, north-west and eastern parts of India and Pakistan also, although understood and spoken occasionally in the rest of India. Urdu is arisen in the 10th century A.D. due to occupation relations, ethnic exchanges, relocations, and military expeditions. Urdu in India is basically developed in close contact with Persian, which was the language of administration and education during the period of Muslim rule.

Resource Development Challenges

Language technology presents a profitable and socially beneficial solution to overcome language barriers. However, the digitization of language data poses challenges, particularly in developing accurate language models for complex languages like Urdu. The first step in enabling computers to understand human language is encoding, achieved through the Unicode scheme, which assigns unique codes for Urdu characters.

Language Resources

Numerous materials exist to analyze Urdu data and develop technologies. These resources include text and speech corpora, dictionaries, ontologies, and multimedia databases, alongside software for collection, preparation, and analysis. A linguistic corpus, which represents real-time language usage, is vital for developing various language technologies.

The main application areas of language technology include spell and grammar checking, speech recognition and synthesis, machine translation, and information retrieval. Language resources are foundational for these tools, enhancing communication and interaction between humans and computers.

Prominent Institutions and Initiatives

Linguistic Data Consortium for Indian Languages (LDC-IL)

LDC-IL, housed within the Central Institute of Indian Languages, has developed a comprehensive Urdu text and speech corpus from various sources, including books and newspapers. The data covers multiple domains, with significant contributions to language processing.

AI4Bharat

AI4Bharat, a research lab at IIT Madras, is committed to enhancing AI technology for Indian languages through open-source initiatives. The lab has developed and released an extensive set of datasets, tools, and cutting-edge models. Its focus areas are transliteration, natural language understanding and generation, translation, automatic speech recognition, and speech synthesis.

Indian Languages Corpora Initiative (ILCI)

The ILCI - Indian Languages Corpora Initiative has been a welcome move by TDIL to develop national corpora based on national standard. The phase-1 saw the development of parallel annotated corpora in 12 major Indian Languages including English in India's national standard in POS annotation. The size of this corpora project finishing in Feb 2012 is 600,000 annotated sentences with each sentence having an average of 16 words (9600,000 annotated words). The phase-2 purposes to include 4 north eastern languages (Nepali, Bodo, Assamese, Manipuri) and Kannada and add 11,00,000 new sentences (500,000 sentences in 5 new languages including the 4 languages of NE and 600,000 in 12 existing languages). The total size of the corpora after phase 2 is estimated to be approximately 27 million parallel annotated and chunked words in the domain such as: Health and Tourism (HT), Agriculture and Entertainment (AGENT)

UMC005: English-Urdu Parallel Corpus

UMC005 English-Urdu is a parallel corpus of texts in English and Urdu language with sentence alignments. The corpus can be used for experiments with statistical machine translation. The texts come from four different sources: Quran Bible Penn Treebank (Wall Street Journal) Emille corpus We provide the religious texts of Quran and Bible for direct download. Because of licensing reasons, Penn and Emille texts cannot be redistributed freely. However, if you already hold a license for the original corpora, we are able to provide scripts that will recreate our data on your disk. Our modifications include but are not limited to the following: Correction of Urdu translations and manual sentence alignment of the Emille texts. Manually corrected sentence alignment of the other corpora. Our data split (training-development-test) so that our published experiments can be reproduced. Tokenization (optional, but needed to reproduce our experiments). Normalization (optional) of e.g. European vs. Urdu numerals, European vs. Urdu punctuation, removal of Urdu diacritics.

OPUS

A growing collection of translated texts, aligned as a parallel corpus for easy access.

Kaggle

A platform for sharing and discovering datasets, including Urdu speech datasets.

Provides handwritten data from native Urdu writers.

FutureBeeAI

provides 2000+ datasets for AI development process.

ULCA

Universal Language Contribution APIs (ULCA) is an open-source, scalable data platform that supports a variety of datasets for Indic languages. It also provides a user-friendly interface for interacting with these datasets.

MANUU

The "MANUU: Handwritten Urdu OCR Dataset" is an extensive and meticulously curated collection to advance OCR (Optical Character Recognition) for handwritten Urdu letters, digits, and words. The compilation of the dataset has been conducted methodically, ensuring that it encompasses a wide variety of handwritten instances. This comprehensive collection enables the construction and assessment of strong models for Optical Character Recognition (OCR) systems specifically designed for the complexities of the Urdu script. Ensuring public accessibility of this resource is of utmost importance. The website in question is accessible to the general public. The information was generated through a collaborative endeavor that engaged the participation of more than 600 writers. The presence of a varied group of contributors guarantees a wide range of handwriting styles, variances, and subtleties, leading to a dataset that closely resembles the authentic diversity found in handwritten Urdu literature.



The resources available for Urdu

Resource Resource Centre Specification Source
Text corpus LDC-IL 51,61,927 words View Resource
AI4Bharat 1,01,110 sentences View Resource
OPUS 25,377,417 sentences View Resource
Synset MeitY Approx 34,347 synsets View Resource
Speech corpus LDC-IL 99:18:21 hours Raw Speech View Resource
LDC-IL 50:09:56 hours Sentence Alligned View Resource
Kaggle 2500 Wave Files View Resource
FutureBeeAI 60 Speech Hours View Resource
ULCA 439.11 hours View Resource
TTS AI4Bharat 78.61 hours View Resource
OCR MANUU Handwritten Urdu OCR Dataset View Resource

Conclusion

The development of language resources for Urdu is a collaborative effort involving various institutions and initiatives. These resources not only support the advancement of language technology but also preserve and promote the rich linguistic heritage of Urdu.