Established in 2007, the Linguistic Data Consortium for Indian Languages (LDC-IL) is a scheme of the Department of Higher Education, Ministry of Human Resource and Development, Government of India implemented by and housed inside the Central Institute of Indian Languages, Mysore.
Currently fully funded by the Government of India, as the name suggests, the Consortium is expected to generate its own funds and become a self-sufficient Institution in itself by way of developing the resources and distributing them to the interested developers, researchers and organizations engaged in using such resources.
LDC-IL has started distributing linguistic resources for Artificial Intelligence (AI) and Natural Language Processing (NLP), mainly in Indian languages, since 4th April, 2019 through its Data Distribution Portal when the portal was launched by the Hon'ble Vice President, Shri Venkaiah Naidu.
Language data is the key ingredient in terms of research and development in the area of language technology. As the time goes by, an increasing number of researchers are seeing the potential benefits of the use of an electronic corpus as a source of empirical language data for their research. The issues surrounding collection, processing and annotation of the quantities of linguistic data make it necessary to involve a number of disciplines like linguistics, computer science, statistics, engineering etc. Corpus linguists, as we all know, often use computational methods when analyzing their data whereas the computational linguists are dependent on computer-readable linguistic data to use in their research and in building practical tools and programmes.
In the founding years of HP Labs India (HPL India) in 2002, the lab had a Department of Language Technology and Applications to research and address language barriers to adoption of Information and Communication Technologies in countries like India. Prof. A.G. Ramakrishnan, Dept. of Electrical Engineering, Indian Institute of Science (IISc), Bangalore, was then working as a Principal Research Scientist with this department, on leave from IISc. The research team at HPL India felt that the lack of publicly available high-quality linguistic data in Indian languages was a major barrier to research in this area. They proposed the idea of initiating a major data collection activity to aid research in technology for Indian languages, similar to the Linguistic Data Consortium (LDC) at the University of Pennsylvania. This idea was supported by Dr. S. Ramani and Dr. Gita Gopal, the then Director and Associate Director respectively of HP Labs India. This was later also enthusiastically endorsed by Dr. Kris Halvorsen who was the Center Director at HP Labs, Palo Alto.
HP Labs India felt that, while they will actively contribute to the creation, this effort would be best spearheaded by an Indian national institution. It was felt that the Government of India would offer some form of support to this activity, because of its significance to national languages. The initial draft proposal for the creation of a Consortium was put together by Prof. A.G. Ramakrishnan and Ms. Kalika Bali with contributions from others in the Department such as Dr Sriganesh Madhvanath, Dr Sitaram Ramachandrula and Dr K.S.R. Anjaneyulu. Dr. S Ramani suggested that Dr. Rajeev Sangal, Director of IIIT, Hyderabad could be the Principal Investigator of the project, being a senior researcher in the language technology arena. Accordingly, a joint proposal was finalized. In addition to Prof. A G Ramakrishnan, Prof. Pushpak Bhattacharya from the Indian Institute of Technology (IIT), Bombay, and Prof. B Yegnanarayana from IIT Madras agreed to serve as co-investigators of the proposal. Finally, as per the suggestion of Prof. Rajeev Sangal, Dr. Uday Narayan Singh, then Director of CIIL, was also added as a co-investigator.
HP Labs India suggested convening a meeting involving all the major researchers in India in language technology to decide the various aspects of the linguistic data collection prior to submitting this proposal to the Government of India. The venue for this meeting was chosen to be CIIL, Mysore and HP Labs India fully funded the meeting in the second week of August 2003. HP Labs India invited Prof. Aravind Joshi and Dr. Mark Liberman, Director of LDC, University of Pennsylvania, for the meeting at Mysore at HP Labs India’s expense. A follow-up meeting was conducted at IISc, Bangalore with a number of key researchers, including Prof. N. Balakrishnan, Chairman of Supercomputer Research Center, IISc, where Prof. A G Ramakrishnan summarized the discussions at the Mysore meeting. Subsequently, the proposal was finalized and submitted to MHRD by Prof. Uday Narayan Singh.
This consortium, being set up on the lines of the LDC at the University of Pennsylvannia (USA), does not only create and manage large Indian languages databases, it also provides a forum for researchers in India and other countries working on Indian languages to publish and build products for use based on such databases that would not otherwise be possible.
LDC-IL is: