TikTok Miner turns a creator's whole catalog into something you can ask questions of
I built an ETL pipeline that pulls a creator's whole catalog with its play counts, transcribes the videos I pick, then chunks and embeds everything into a vector store, the “Digital Brain” you can query.
Studying one creator: three hours of scrolling, thirty videos, no pattern
Every short-form script I wrote started the same way: open a creator who was winning, watch at 2x, pause, retype the hook into a doc, screenshot the caption, scroll on. An evening of that produced notes on maybe thirty videos and a hunch about a pattern I could never check.
The accounts I most wanted to study made it worse. The sharpest playbooks in my niches lived on Douyin, in Chinese, behind catalogs of hundreds of videos. One question mattered: which openings does this creator repeat, and do the repeated ones earn the plays? A feed cannot answer that. A feed shows you videos one at a time, in the order it chooses, in whatever language they were shot. Somewhere around the hundredth swipe of one catalog I admitted the problem was not my patience. The catalog should be queryable, and no amount of disciplined note-keeping was going to make it so.
Research and production are two different tools
Miner is the research half. It embeds a whole catalog into a vector store and answers questions put to it, which is the shape I want when I am working out how someone keeps winning.
Hyper Creator is the production half. It starts from an idea I have already chosen and moves it through language, rewrite and voice until a publishable file exists. I open Miner to study a catalog, and I open Hyper Creator when one of those answers has to become a deliverable. They meet at the transcript, and each one stops where the other begins.
Creative Center, downloader chains, a spreadsheet
TikTok's own Creative Center is good at what it does: trending sounds, hashtags and top-performing ads, refreshed constantly. But trends were never my question. My question was about one specific creator's body of work.
The manual route existed too: a downloader tab, a transcription app, a spreadsheet. Chained together they could turn one video into text in about ten minutes of copy-pasting, and the play counts stayed behind on the platform. Multiplied across a catalog, that was the old evening again with more tabs open. And nothing off the shelf spoke Douyin at all, where login walls and anti-bot defenses stop casual tools cold. What I needed was narrower than any of it: pull one creator's whole catalog with the stats attached, transcribe exactly the videos I choose, verbatim, in the original language, and keep a person between the scrape and the model at every step.
Narrower than any tool on the market, on purpose
I fixed the boundary before writing anything. This would not become a scheduler or an auto-poster. It stays one pipeline with two human gates and clean exports.
Paste a creator URL and get the catalog back as a sortable table, with plays, likes, comments and shares attached. Keep every transcript verbatim and in its original language, Chinese included. Never let the model run unattended: I pick the videos worth transcribing, and I sign off the transcript before anything gets rewritten.
The transcription API I never had to add
I assumed I would need a dedicated speech-to-text service, then tried uploading the audio straight to Gemini 2.5 Flash through the Files API with a verbatim-transcript prompt. It came back clean, in the right language, Chinese included, in one call: recognition, language handling and formatting collapsed into a single request, and the file is deleted right after. ffmpeg strips each video to a 64kbps mp3 first, so the upload stays small.
Collection was the opposite: nothing collapsed, everything layered. I built it to behave like the person whose account is reading, because that is what the account is entitled to see. So the collector runs slowly on purpose. It waits a randomized interval between pages instead of a fixed one, walks the catalog by simulating the clicks a reader would make, and signs in with the user's own session. There is more in there, and I am not going to write the whole recipe down. The constraint is simple: one person's own access, at one person's pace. The proxy that makes it reachable from the browser is pinned to douyin.com and tiktok.com and refuses private IPs, so it cannot be turned into an open relay. Everything lives in one 2,400-line App.tsx workbench on an Express backend, with Stripe credits, free 20, lite 200, pro 1,500, enforced on both client and server.
React 19
Vite
Express
Gemini 2.5 Flash
Playwright
Vercel
Paste a URL, get a catalog you can interrogate
An embedded catalog opens reads that scrolling never allowed, and the ones below sit at different stages of proof.
I put a question to one creator's corpus, like how this person frames pricing, and the retrieval-augmented chat answers from hundreds of their own videos, with the clips it drew on.
I read an engagement heatmap per video and set two creators side by side, so a format difference shows up as a shape. That score is mine, and I have not validated it.
The workers can re-run and pick up new posts as they drop, so a tracked creator stays current without anyone remembering to check. That schedule is built and I have not run it yet.
Looking across the few creators in this instance for the topics heating up in a niche is the read a content calendar needs. That one is built and I have not run it yet.
I split the models on cost: Gemini runs the reasoning passes, DeepSeek reads the code and logic, and Pinecone holds the embeddings behind a React and TypeScript front end. I designed it so each account's vector store stays separate, because one shared index would be the failure I could never walk back.
The evening of scrolling became a query
Studying a creator used to be an evening of scrolling that logged thirty-ish videos, against the 2,050 already in this instance. Now one pasted URL brings the whole catalog back as a table I sort by plays, and any video I select returns as a verbatim transcript minutes later. I will not invent a revenue number for this one. What I can measure is the catalog one paste returns and the transcript I signed off on.
The other change is the language wall. A Douyin video I could not have used at all now goes from Chinese audio to a transcript I have read and signed off, in one sitting. The two human gates are why I trust the output: the model transcribes what I picked and rewrites what I approved, nothing else. Carrying that signed transcript across the language and into a finished script is Hyper Creator, the production half of this pair.
An ETL pipeline with a vector store at the end of it
This is the only tool in the set with a vector store, because it is the only one whose question is open-ended. The others match against a bounded rulebook, where retrieval would add cost and remove certainty.
Application
React 19Dashboard, mining queue and the digital-brain views.
TypeScriptAcross the app and the worker contracts.
ViteBuild tooling.
ExpressThe job API behind the miners.
Pipeline
PlaywrightCaptures the session, walks the catalog.
FFmpegStrips audio before transcription.
Models
Gemini 2.5 FlashTranscription and the chunk summaries.
DeepSeekLogic and comparison passes across creators.
GPT-4oThe retrieval-augmented chat over a creator’s corpus.
Data and delivery
Neon PostgresCatalogs, transcripts and job state.
PineconeThe embeddings behind the digital brain and its retrieval.
StripePlan tiers, and nobody has paid for one.
VercelHosting and the serverless ETL workers.
Want this kind of tooling on your team?
This one turns a creator's whole catalog into something you can question.