Reflection on System Efficiency at Meta: A Personal Perspective

One of the most renowned computer programmers of our time, John Carmack, recently published a post about his decision to leave Meta. In his note, he highlighted his concerns about the company’s ineffectiveness in terms of organization and inefficient use of resources. As a former engineer who worked on improving infrastructure efficiency at Meta, I can relate to Carmack’s thoughts and have my own experiences to share.

Photo by Taylor Vick, on Unsplash

At Meta, we hired some of the best and brightest engineers from around the world and compensated them highly. We had more than 50k engineers and even built and maintained our own data center. Despite these resources, our systems were not running at the highest level of efficiency. For example, Carmack mentioned that production systems were only using 5% of their GPU. This is unacceptable for a company with such a talented team and vast resources.

What went wrong?

There were a few issues that contributed to this inefficiency:

• Lack of accountability and meaningful metrics: Many of the efficiency metrics at Meta were superficial and did not truly reflect the important work being done. Instead of focusing on a few key metrics, we kept adding more and more without much consideration. Even for the most important metrics, we would often exclude certain parts of the system that has no clear ownership, rather than finding a solution.
• Short-term thinking by engineers: I have seen and heard of many cases where “senior” engineers would quickly build something that didn’t work, use some “metrics” to claim promotion, and then move on to the next “big” thing. Leaving behind a buggy system that required junior engineers to fix and maintain, setting a poor example for the team.
• Lack of honesty and long-term thinking: Overall, we were not being honest with ourselves about the real problems at the company. We focused on things that didn’t matter and did not incentivize engineers to think long-term.

I, too, was part of this problem during my time at Meta. I tried to raise concerns and question these metrics, but eventually I learned to go with the flow and compromise. I remember sitting in capacity planning meetings where a select group of people made decisions on how many billions of dollars we would spend on building new infrastructure and which teams would get the biggest slice of the pie. Initially, we forecasted capacity and ran the numbers honestly, but then resources were allocated based on the influence of managers rather than the importance of the work. I was there to make everyone look good, not to challenge the status quo. Eventually, I became the type of person I despised, which is why I decided to leave.

What have changed?

Since joining Quora, a smaller company with only 100 engineers, I have seen firsthand the benefits of a more efficient approach to resource allocation. At Quora, we prioritize the efficient use of resources in order to remain competitive and fulfill our mission of sharing and growing knowledge. In my current role, I am trusted to make decisions on how to best utilize resources and I am damn making sure they are being used effectively. Here are a few things I have done differently:

• Set clear goals: To improve the efficiency of the system, I have prioritized establishing clear goals and avoiding metrics that do not provide meaningful information. Rather than tracking numerous metrics, I have focused on a select few that accurately reflect the system’s performance. If you’re interested in learning more about how to choose the right goals and avoid vanity metrics, I recommend checking out this post by Julian Shapiro. It has some great tips and insights on the subject.
• Maintain a clear purpose: Having a clear sense of purpose and values makes it easier to make difficult decisions. It is unfortunate to see mass layoffs in the tech industry, especially when they affect friends and colleagues. As someone responsible for optimizing systems, I care deeply about efficiency. If I see a colleague being let go due to the company’s poor capital efficiency, I take it personally. It is my mission to ensure that the company allocates resources on important things and uses them effectively in order to succeed in the long run.
• Learn to inspire others: As an existentialist, I believe that actions speak louder than words. To inspire others and work towards a common goal, it is important to lead by example. This means demonstrating a strong level of care and commitment, sharing your vision and strategies, communicating effectively, regularly checking in, recognizing small contributions, and celebrating each milestone. All of these actions can help motivate and inspire others to strive for success.
• Be honest and don’t compromise: Over the years, I have grown and am now capable of speaking up and challenging the status quo if I believe it will lead to a more efficient and effective use of resources. I believe that it is important to remain true to our values and not compromise on the things that matter most to our company.

Our efforts to improve system efficiency have paid off, resulting in a 20% reduction in infrastructure costs in the past quarter. This success has allowed us to continue hiring and investing in new technologies, such as the beta testing of Poe, a “Platform for Open Exploration” for Large Language models. While other companies have had to resort to mass layoffs and scale back on innovation, we have been able to maintain our resources and focus on long-term growth. Efficient operations are essential for any company looking to thrive in the face of competition and market changes.

Closing thought

Despite any challenges, I am still optimistic about the future of Meta and have chosen to retain 100% of my RSU vesting from the company. My time at Meta has been extremely valuable, as it has provided me with valuable insights on system efficiency and introduced me to many skilled and impressive individuals. Like John Carmark, I wearied of the fight, but I have faith in the team members who are constantly working to improve the efficiency of Meta’s system. I always have confidence in them and wish them all the best of luck.