Building kannada-kasturi-embeddings

Embeddings are numerical representations of real-world objects, such as words, phrases, text, images, audio, and video. Since the real world is so complex, these representations are usually vector arrays of floating-point numbers. This helps computers process meaning, context, and semantic relationships using distances and directions between vectors.

Word2Vec and FastText

Embeddings are learned numerical representations. The actual values depend on the model architecture, training data, and its training objective. So the same object (say a word) gets different embeddings in Word2Vec and GPT. Think of an embedding as a model’s custom learned coordinate system for meaning.

This coordinate system lets us do useful things. Find semantically similar items, power search, feed downstream ML models, or even perform arithmetic on meaning. A general example from Word2Vec is king − man + woman ≈ queen. There are mainly two types

• SkipGram: Where the model predicts context words given a target word
• CBOW (Continuous Bag of Words): Where the model predicts the target word from its context.

I wanted to build a simple Word2Vec model for Kannada just to try it out and see if it’s useful. In the process I also built a FastText model. FastText is the same as Word2Vec with character n-grams. By default, I have built Skipgram based models. But you can try CBOW too.

With n-grams support, FastText makes it easy to handle out of vocabulary words and capture subword information, which Word2vec is not good at. For example, given the settings minn=3, maxn=8, FastText breaks each word into character n-grams of lengths 3 to 8 and learns embeddings for those n-grams as well. There is also a wordNgrams setting, set to 2 by default. We can play with this setting. wordNgrams is used to learn embeddings about the sequences of adjacent words.

For example consider the sentence:

"ಬೆಂಗಳೂರು ಕರ್ನಾಟಕದ ರಾಜಧಾನಿ"

With:

wordNgrams=1 (default) Learns only individual words:

ಬೆಂಗಳೂರು
ಕರ್ನಾಟಕದ
ರಾಜಧಾನಿ

wordNgrams=2 Also learns bigrams:

ಬೆಂಗಳೂರು ಕರ್ನಾಟಕದ
ಕರ್ನಾಟಕದ ರಾಜಧಾನಿ

wordNgrams=3 Also learns trigrams:

ಬೆಂಗಳೂರು ಕರ್ನಾಟಕದ ರಾಜಧಾನಿ

Data

For the corpus, I went with the Kannada text of Kasturi Magazine OCRed by Sanchaya and available on Archive.org. My repository has this text; you can use it instead of redownloading from Archive.org. I chose Kasturi content because it’s the most generic Kannada content available, small but good enough to do quick tests. It’s a monthly family magazine published from the house of Samyukta Karnataka, a very popular newspaper. You can think of it as Reader’s Digest of Kannada. I have not given the script used to download, as it’s not required. But it’s not that difficult to write if you want to. My bots were identified by headers={"User-Agent": "thejeshgn.com/1.0"} and all the downloaded files are named kananda_kasturi_{identifier}.txt where identifier is archive..org identifier.

Build and Test

You can run the trainer, change parameters and test them. I have included all the code and steps for both Fastext and Word2Vec.

Project is available on Codeberg, Github and HF.

# Prepare the clean corpus.txt from the raw inputs
uv run prepare_corpus.py prepare ../data/ ./corpus.txt

# Train the embeddings with default parameters 
uv run fasttext_embeddings.py train ./corpus.txt --recursive --out ../models/kasturi_fasttext_v4

# Try the embeddings
uv run fasttext_embeddings.py test ../models/kasturi_fasttext_v4.bin --word ಕನ್ನಡ
uv run fasttext_embeddings.py test ../models/kasturi_fasttext_v4.bin --word ರಾಜಧಾನಿ
uv run fasttext_embeddings.py test ../models/kasturi_fasttext_v4.bin --sentence "ಬೆಂಗಳೂರು ಕರ್ನಾಟಕದ ರಾಜಧಾನಿ"
uv run fasttext_embeddings.py compare ../models/kasturi_fasttext_v4.bin

For comparison and evaluation, I used the same method that I had used before for testing other embeddings. Take a bunch of Kannada words (I have used the same bunch), get the cosine similarity between them and create a comparison matrix. I also compared my matrix against the word vectors trained by FastText team using Wikipedia and Common Crawl data. You can see them below.

TODO

I have called it kannada-kasturi-embeddings because it’s a good name and we started with Kasturi. But it wont be limited to Kasturi magazine data alone. I will add other newspaper and magazine data to it and build a comprehensive embedding that’s useful. I will also keep comparing to see how it performs. The current test and compare methods are very basic. I will come up with a better method. So the TODO list is

• Add more quality data
• Improve the clean_text function
• Add more automated quality testing and comparisons
• Build different embedding models and not just Word2Vec or Fastext

In the meantime, try it and let me know what you think.