Key Idea Human validation is not only for evaluating an LLM. It can also calibrate how the LLM is used as a scalable measurement instrument for population estimation. An LLM can classify thousands of records at low cost, but the proportion it classifies as positive is not necessarily the true propor...

Introduction As Large Language Models (LLMs) move from research prototypes to production systems, the developers of these systems need rigorous performance evaluation. In particular, we need confidence intervals around estimates of system accuracy. However, LLMs introduce a challenge that is unusual...

Authors : Marsan Ma, Nikhil Lopes, Raj Amrit, Hong Lu, Dipankar Biswas, Trent Kyono Leadership: Iris Wang, Madhu Kurup Recommendation and ranking systems power many of the most important experiences on large internet platforms. Yet the models that run in production are rarely the largest models we c...

Why agentic coding tools demand a new identity for software engineers After more than two decades of professional software engineering, I have arrived at a set of conclusions that I find very uncomfortable. The era of mostly manual coding has ended. IDEs, in their current form, are no longer necessa...

At Indeed we’ve open sourced a new React Native repository which makes it simple to measure Lighthouse scores in your mobile apps. We think it will help other organizations better measure their app performance, especially for companies similar to Indeed who are transitioning from a web-first to an a...

Introduction At Indeed, our mission is to help people get jobs. We connect job seekers with their next career opportunities and assist employers in finding the ideal candidates. This makes matching a fundamental problem in the products we develop.  The Ranking Models team is responsible for building...

Every so often, the web changes in a way that rewires how we live. In the early days, Web 1.0 let us read. It was a window into information — static pages, digital brochures, news sites. We were spectators peering into a new world. Then came Web 2.0, and we learned to write. We didn’t […]

Here at Indeed, our mission is to help people get jobs. Indeed is the #1 job site in the world with over 580M+ Job Seeker Profiles. For Indeed’s Engineering Platform teams, we have a slightly different motto: “We help people to help people get jobs”. As part of a data-driven engineering culture that...

Goal This blog is for engineering teams, architects, and leaders responsible for defining and implementing a workload identity platform and access controls rooted in Zero Trust principles to mitigate the risks from compromised services. It is relevant for companies using Kubernetes to manage workloa...

In the past, Indeed has used a variety of metrics to evaluate our client-side performance, but we’ve tended to focus on one at a time. Traditionally, we chose a single performance metric and used it as the measuring stick for whether we were improving or degrading the user experience.  This made it ...

Have you ever wanted a crystal ball that would predict the best A/B test to boost your product’s growth, or identify which part of your UI drives a target metric? With a statistical model and a SHAP decision plot, you can identify impactful A/B test ideas in bulk. The Indeed Interview team used this...

Photo by Kevin Ku on Unsplash Indeed adds over 30 million jobs online every month, which helps connect 250 million job seekers to prospective employers. How do we keep our services available, fast, and scalable? That’s the ongoing challenge for our site reliability engineering (SRE) team. What is SR...

Photo by Jonathan Chng on Unsplash Over the last few years at Indeed, we noticed our public-facing web applications were loading more slowly. We tested numerous ways to improve performance. Some were very successful, others were not. We improved loading speeds by 40% but we also learned that speed i...