Senior Machine Learning Engineer, Pegasus

Job description

Who we are

Video is 90% of the world's data. Most of it is invisible to machines.

TwelveLabs builds the intelligence layer to change that. Our multimodal AI models understand video the way humans do — across sight, sound, and motion — and power production-scale AI workloads across media, entertainment, sports, security, and government.

We have raised more than $210 million from NEA, Radical Ventures, Amazon, NVIDIA, Snowflake, Databricks, Index Ventures, NAVER Ventures, Korea Investment Partners, Quadrille Capital, Red Bull Ventures, and AI pioneers including Fei-Fei Li, Silvio Savarese, and Alexandr Wang.

We are a global company, headquartered in San Francisco with offices in Seoul, New York, and London, and employees around the world. We believe the differences in our cultural, educational, and life experiences make our products stronger. Building technology that understands the world in all its complexity requires people who see it from every angle. We are looking for individuals who are driven by hard problems and want their work to matter. Come build it with us!

About Jockey

Jockey is TwelveLabs' unified agentic system that reasons across your videos and images. It combines a reasoning model with a memory layer that builds a knowledge store from your corpus.

No context window holds a video archive. We work at a million hours of video. A single model forward pass can tell you about one file; it can't reason across a corpus, and no context window closes that gap. Jockey decomposes a query, retrieves, segments, and reasons across thousands of videos and images. Point it at an archive, ask for a highlight reel or the best viral moments, and it returns timestamped cuts you can use. Corpus-level understanding you can act on is the whole product.

Built for agents, not just people. As AI agents increasingly become the primary consumers of video, we're building production-grade infrastructure that scales to millions of hours while delivering reliable, high-quality results for both human users and autonomous agents.

We build on models we own. Marengo, our embedding model, resolves a query like "the moment we almost missed the flight" into real retrieval. Pegasus, our video-language model, returns structured, timestamped moments on a schema you define. We ship and improve both continuously, so Jockey's quality compounds with every release — no re-integration for customers. Few teams get to build an agent on a stack they control end to end.

Deep expertise, one system, open culture. Foundation models, knowledge construction, search, and the agent harness all live in one org. Each team owns its domain and is expected to have deep expertise in it — but like a Formula 1 team, we optimize for the global system, not local parts. A model gain that doesn't expand what the agent can do isn't a gain. We trace a single algorithm change through to end-system behavior, and share work in progress weekly, not just finished results. Anyone can pull the context they need from any team.

About the team

The Cognition Models team owns the models that turn video into structured understanding and reasoning: Pegasus, our video-language model, and Jockey Core, the reasoning LLM behind Jockey. In the model stack we sit between Perception Models (embeddings and retrieval) and the agent system — taking what's retrieved and producing structured understanding and the reasoning to act on it.

We focus on multimodal systems with high instruction-following capability and complex, hierarchically structured outputs. Our work spans training infrastructure from pre-training to RL, temporal segmentation and structured metadata extraction, large-scale inference and serving systems, data-curation and evaluation pipelines, and building Jockey Core. We ship products with real-world value rather than doing research in isolation, working as a goal-oriented, cross-functional team of ML researchers and engineers — using the most advanced compute in the world, including NVIDIA B300s, to accelerate the research-to-production cycle.

About Pegasus

Pegasus is TwelveLabs' video-language model — it turns video into useful analysis by reasoning over visuals, speech, audio, and on-screen text. A key capability is Segment, our time-based metadata feature: instead of a broad question about a video, customers define the exact segment types they care about and the metadata fields they want back, and Pegasus finds the relevant start and end times and returns structured metadata for each segment — titles, summaries, topics, people, visual subjects, confidence, or domain-specific labels. This turns video into time-based, structured data that flows directly into search, archive, editing, compliance, or content-management workflows.

In this role, you will

• Build, improve, and operate production ML systems for Pegasus, with a focus on reliability, performance, and maintainability.

• Work across core parts of the ML stack, including deployment, inference, evaluation, monitoring, and supporting infrastructure.

• Develop systems for serving Video Language Models (VLMs) and handling multimodal data and metadata at production quality.

• Make strong technical decisions within your area and drive execution with a high degree of ownership.

• Explore and adopt AI-assisted development tools such as Claude, Gemini, and GPT to improve productivity across coding, experimentation, debugging, and documentation.

You may be a good fit if you have

• Strong software engineering and machine learning fundamentals.

• Experience building and shipping ML systems in production.

• Experience with multimodal data and familiarity with areas such as computer vision, natural language processing, LLMs, or VLMs.

• Experience with distributed ML or data workflows, ideally in Kubernetes-based environments.

• Strong engineering judgment around performance, reliability, and maintainability in production environments.

Preferred qualifications

• Experience serving or optimizing LLM/VLM systems in production.

• Experience with inference optimization techniques such as batching, caching, or quantization.

• Experience building AI/ML systems from early-stage development through production deployment.

• Master's or PhD in Machine Learning, Computer Science, or a related technical field.

Read more about the team

• There are few companies in the world that are truly serious about video.

• Even an outstanding model ends at ‘impressive’ if it isn’t used.

• The people who built Pegasus 1.5

• Turning video into structured assets: Building the Time-Based Metadata (TBM) pipeline

• Cutting Edge Isn’t Plug-and-Play: Customizing FlashAttention-4 on B300

Hiring Process

Application review → Recruiter interview → Hiring Manager interview → Job interview (1–2 rounds) → Final interview → Reference check and offer → Final acceptance

*The hiring process may vary by role and may change depending on schedule and circumstances. Please note that the job interview may include an in-person interview.

Benefits and Perks

Growth & Tools

• A Global Team that grows together with global B2B customers

• Hybrid work with both autonomy and collaboration

• Latest MacBook and work-from-home equipment worth KRW 700,000, with latest equipment replacement every 3 years

• Unlimited LLM tokens so the Tech job family can use AI efficiently and responsibly

• Annual professional development budget worth KRW 1.4 million usable for courses, conferences, memberships, etc.

• English education program and global buddy program

• Taxi fare support for commuting at night and on weekends

Meal & Snack

• Corporate card worth KRW 7.2 million per year that can be used freely for meals, transportation, etc.

• Snack bar in the office (snacks, coffee, seasonal fruit, etc.)

• When working in the office, dinner meal allowance provided after 7 PM

Wellness & Family

• Annual health checkup for yourself and one family member

• Group insurance enrollment (choose 1 of accident insurance / dental insurance / family accident insurance)

• Flu vaccination cost support

• Paid Holiday Break for 2 weeks at year-end

Additional Notes

• Common qualification for all roles in Korea: No grounds for disqualification from overseas travel

• All new hires are subject to a 3-month probation period. Salary is paid at 100% during that period.

• If false information in documents submitted during the hiring process or fraudulent conduct is confirmed, the offer may be canceled or future hiring may be restricted.

If someone contacted you about this role and you’d like to verify they’re a member of our team, please email [email protected]