Upload a raw screen recording. Framewise watches it, decides what to cut and where to zoom, writes the narration, casts a voice and renders a finished demo video in 11 languages.
Shipping a product demo takes a chain of people. Someone writes the script, an editor cuts the dead air and adds zooms, someone records the voiceover, and a localization vendor repeats it all for every language. That's days of work, per video, per language.
Features in action
Running captions
Each word highlights as it's spoken.
Animated zoom
Eases in on the exact UI element in use.
Background music
MusicGen bed that ducks under the voice.
What it does
Understands the recording. Builds a timestamped log of every click, scroll and UI change.
Edits, not just narrates. Cuts dead air, speed-ramps repetitive stretches and adds animated zooms aimed at the exact UI element in use.
Writes and voices the narration in 11 languages, with an LLM casting the speaker from about 11,000 voices.
Speaks like a real tech creator. Technical terms stay in English inside Tamil or Hindi narration.
Finishes the video. Word-level captions, click ripples, a branded backdrop and locally generated background music.
How it works
Eight agents orchestrated with LangGraph, sharing one state:
Frame Extractor samples frames with FFmpeg.
Cursor Detector is folded into vision analysis.
Vision uses Gemini to watch the video in 60s windows and log every action.
Audio transcribes existing narration and detects silence.
Analysis merges vision and audio into one timeline tagged cut, zoom or keep.
Script Planner writes the narration and the concrete edit plan.
Music generates an intro sting and a looping bed with MusicGen.
Theme and Voice picks a backdrop from the app's colours and casts the voice.
The render engine then applies cuts, speed ramps, zooms, click effects, narration, music and captions, and renders twice, once per voice engine (ElevenLabs and Sarvam), so you pick the better take.
Why 60-second windows?
A single Gemini request over a long video only described its first minute. It returns a fixed budget of about 20 to 25 events regardless of length, and still reports success. Overlapping 60s windows give each stretch its own budget: 64 events across 229 seconds, with the largest gap at 15s.
Languages
English, Tamil, Hindi, Telugu, Bengali, Kannada, Malayalam, Marathi, Gujarati, Punjabi and Odia. Indian languages use Sarvam's native TTS rather than a Western model approximating them.
Tech stack
AI
Gemini, ElevenLabs, Sarvam Bulbul v3, MusicGen
Video
FFmpeg, libass, libx264, MoviePy, pydub
Backend
LangGraph, FastAPI, Pydantic, SQLite, WebSockets
Frontend
React 18, TypeScript, Vite
What's next
The script planner sometimes under-writes long videos, leaving a quiet tail, and Sarvam renders don't yet get word-level captions. Both are on the list, alongside the research paper.
Because nobody says कार्यालय when they mean office.
Text-to-speech for Indian languages that pronounces the English words people actually mix in. A fine-tune of AI4Bharat's IndicF5 on real Hindi-English speech.
मैं आज office जा रहा हूँ, morning में एक important meeting है और उसके बाद team के साथ new project discuss करना है।
6 / 6English words recovered
71%lower English CER
0loss on pure Hindi
8unseen languages improved
The problem
Indians don't speak pure Hindi or pure English. We say office, meeting and project in the middle of Hindi sentences. IndicF5 reads those English words with Indic phonetics and turns them into gibberish: office came out as अफे, important meeting as टहौर का निये.
Voice cloning
Record about 10 seconds of your voice, and the model speaks any Hinglish sentence in that voice. Here it is cloning mine.
Reference (my real voice)
My voice, about 10s
0:00
Generated (voice cloned)
Cloned, new sentence
0:00
कल सुबह मेरी client के साथ call है, उसके बाद presentation ready करनी है।
Also ships with 10 built-in voices: Hinglish, Tamil-Hinglish, Bengali, Gujarati, Kannada, Malayalam, Marathi, Odia, Punjabi and Telugu.
How I fixed it
Adapter-style QLoRA training on a frozen 0.3B IndicF5 backbone, using real Hindi-English speech from OpenSLR-104, sized to fit a single Kaggle session.
Model
Base
ai4bharat/IndicF5
Architecture
F5-TTS DiT, 0.3B params
Transformer
dim 1024 · depth 22 · 16 heads
Text encoder
text dim 512 · 4 conv layers · FF mult 2
Vocabulary
2,546 tokens, unchanged
Audio
Sample rate
24 kHz
Mel spectrogram
100 channels
STFT
n_fft 1024 · hop 256 · window 1024
Vocoder
Vocos
Training
Strategy
Frozen backbone
Trainable layers
Text embeddings, in/out projections, norms, prediction head
Epochs
20
Learning rate
5e-5 · 500 warmup steps
Batch
19,200 frames · grad accumulation 4
Gradient clipping
max norm 1.0
Hardware
2× NVIDIA T4 (32 GB) on Kaggle
Time
About 10 hours
Data and evaluation
Dataset
OpenSLR-104 Hindi-English
ASR judge
Whisper large-v3
Metric
Character error rate (CER) + transcript review
It works beyond Hindi
Trained on Hindi-English only, yet English pronunciation improves in 8 languages it never saw. Same sentence: "I'm going to office today, there's an important meeting in the morning."
Base IndicF5
நான் இன்னைக்கு நாப் ஹேவ் போறேன் மூனேல ஒரு எந்தாண்ட நேயை இருக்கு
This model
நான் இன்னைக்கு அவ்வேச் செல்கிறேன் மார்ணிங்கல ஒரு இம்போர்ட்டன்ட் மீட்டி இருக்கிறது
Base IndicF5
నేను ఇరోజు నాఫె కి వెళ్లుతున్నాను ఉన్నే లో ఒక ఏహటోన ఈ ఉంది
This model
నేను ఇరోజు ఓఫేస్ కి వేలుతున్నాను మోనింగ్ లో ఒక ఇంపోటంట్ మేటింగ్ ఉంది
Base IndicF5
ನಾನು ಇವತ್ತು ನಭಾಯಗೆ ಹೋಗುತ್ತಿದ್ದೇನೆ ಮೋಣೆನಲ್ಲಿ ಒಂದು ಹೋಣಿ ಇಯಿ ಇದೆ
This model
ನಾನು ಇವತ್ ಓಫಿಸ್ಗೆ ಹೋಗುತ್ತಿದ್ದೇನೆ ಮೋರ್ಣಿಂಗ್ ನಲ್ಲಿ ಒಂದು ಇಂಪಾಟಂಟ್ ಮಿಟಿಂಗ್ ಇದೆ
Base IndicF5
मी आज ओफेल आजातोई ओणे मधे एक अहोन इई आहे
This model
मी आज ओफिसला जातोई मौर्णिंग मधेक इंपोर्टेंट मीटिंग आहे
Base IndicF5
હું આજે અફે જઈ રહ્યો છું હોણે માં એક હોર્ચણ એહી છે
This model
હું આજે ઓફેશ જાઈ રહેઓ છું મોર્નિંગમાં એક ઇંપોર્ટન્ટ મેટિંગ છે
Base IndicF5
আমি আজাফে জাছি অনি একটা রিফান্ধ হো নেই আছে
This model
আমি আজ ওফিস জাছি মোনিঙে এ একটা ইম্পর
Base IndicF5
यान इन्न हफले की पोगुनु होरुनेल ओरु टेन्हान मेई उन्ड
And nothing was forgotten: pure Tamil output is character-for-character identical to the base model, and Telugu and Bengali differ by a single character.
Try it
from transformers import AutoModel
import soundfile as sf
model = AutoModel.from_pretrained(
"Tharshan/indicf5_hindi-english_code_switch", trust_remote_code=True)
ref, ref_text = model.voice("ritu_hinglish")
audio, sr = model.generate(
"मैं आज office जा रहा हूँ, morning में एक important meeting है।",
ref_audio=ref, ref_text=ref_text)
sf.write("out.wav", audio, sr)
Self-contained: weights, vocabulary and 10 reference voices ship in the repo. Clone any voice from a short clip and its transcript.
What's next
Pure English is better than base but still drops words on long input, generation handles one or two sentences at a time, and timbre is slightly duller because the training audio came from an ASR corpus. Next up is Tamil-English, trained on natural code-switched speech.
From a topic or a DOI to a podcast you can listen to.
Why
Finding, reading and summarising research papers by hand takes hours, and there's nothing to listen to on the go. I wanted research to be accessible to students and researchers short on time.
What
A multi-agent system that fetches papers, summarises them, groups them by topic, writes a synthesis and narrates it as podcast-style audio with citations. It accepts a topic, a DOI or a PDF.
How
Five agents run in sequence, each handing its output to the next:
Fetcher agent. Searches CrossRef by topic or resolves a DOI, then downloads open-access PDFs through Unpaywall.
Extractor agent. Pulls clean text from each PDF with pdfplumber and PyMuPDF, and summarises it with Llama 3.3 70B.
Classifier agent. Embeds each paper with Sentence Transformers and assigns it a topic using the LLM.
Topic synthesizer agent. Merges all summaries within a topic into one readable synthesis.
Media and citation agent. Formats a citation list and converts each synthesis into an MP3 podcast.
Results
Automated the full research workflow, cutting manual effort by about 70%.
Concise, accurate syntheses across multiple papers.
Ready-to-play MP3 podcasts, plus structured JSON and citations.
General-purpose LLMs give vague, generic answers to everyday banking questions. Customers need clear, specific replies, however they phrase the question.
What
A banking chatbot built by fine-tuning Llama 2-7B chat on banking queries, so it answers in a focused, structured way. The model and dataset are published on Hugging Face.
How
Data. A banking dataset of about 1 million queries and answers, with many phrasings of the same question.
Model. Llama 2-7B chat loaded in 4-bit NF4 with double quantisation, so a 7B model trains on Google Colab.
Fine-tuning. QLoRA adapters (rank 64, alpha 16, dropout 0.1) trained with TRL's SFTTrainer, paged AdamW and a 2e-4 learning rate.
Release. Merged and pushed the fine-tuned model and the dataset to the Hugging Face Hub.
Results
A 7B banking assistant fine-tuned on a single Colab GPU with 4-bit QLoRA.
Consistent answers across different phrasings of the same question.
Open model and dataset on Hugging Face for anyone to reuse.
I build cool things and chase interesting use cases. One year of shipping production-grade AI at BrainSight AI, and my recent studies and work have taken me deep into voice AI and agents.
2nd place, InventAI National-level HackathonWon a cash prize of ₹75,000.
Top 10 finalist, IIT Madras Shaastra'25Selected in the top 10 from 300+ teams.
What I work on
Voice AI and code-switched TTSMulti-agent systems with LangGraphMCP servers and tool integrationRAG pipelinesLLM fine-tuning with QLoRAClinical and medical AI
Currently learning
Voice agentsOptimizing and deploying open-source modelsModel quantizationAgentic system design
Tech stack
Languages and databases
PythonCMySQLMongoDBVector DB
Frameworks
PyTorchTensorFlowKerasHugging FaceLangChainFlask
Libraries
scikit-learnOpenCVPandasNumPy
Tools and hardware
LinuxGitNVIDIA JetsonRaspberry Pi
Experience
AI Development Consultant
BrainSight AI
Jan 2026 – present
Clinician chat agent
ProblemClinicians in hospitals struggled to manage and follow up on patient data and billing. Whenever something failed, they had to chase our support team for answers.
SolutionA chat agent that tells clinicians the live status of their patient data. If a step fails, it explains why, and it helps our team manage all patient data from one place.
How it worksA ReAct-based agentic flow: reason, act, check, repeat until the answer is right. Four agents work together:
Knowledge agent. Hybrid RAG: BM25 + dense vector search, fused with RRF and reranked by a cross-encoder.
DB agent. Picks from a library of pre-approved queries for each tenant and patient. The LLM never writes raw SQL, so data access stays safe and accurate.
Validation agent. Checks every result and loops back for rework until it meets expectations.
Suggestion agent. Recommends the next question based on the user's data and chat history.
B0 field inhomogeneity correction
ProblemB0 field inhomogeneities distort dMRI scans used for tractography and ML diagnosis.
SolutionAn SDC pipeline with SDCFlows, Nipype and Synb0-DisCo v1 (FSL TOPUP + FreeSurfer) that corrects the distortion.
ML Intern
BrainSight AI
Sept 2025 – Dec 2025
Built a RAG-based clinical support chatbot and automated the billing workflow.
Invoice approval automation
ProblemBilling teams manually cross-checked every patient bill and MR form, emailed an acknowledgment for each, and built a consolidated invoice per tenant every month.
SolutionAutomated all three steps: TrOCR-based extraction to verify bills against processed data, per-bill acknowledgment emails, and month-end consolidated invoices with bill proofs.
Freelance Developer
Pingus
2025
Food ordering app. Full-stack app with real-time order tracking, built with React, Node.js and MongoDB.
QR table menu system. QR-code menus with daily specials, an admin panel for menu updates and customer food reviews.
B.Tech, Computer and Communication Engineering
Amrita School of Engineering, Chennai
Expected May 2027
Projects
Live demo
Hindi-English code-switched TTS
Fine-tuned AI4Bharat's IndicF5 on OpenSLR-104 so it can speak sentences that switch between Hindi and English. Public weights and a demo Space.
