Toc
  1. Assessing Your Role
  2. Creating Your Culture
  3. Applying Core Values
  4. Creating Cultural Policy
  5. Writing a Career Ladder
  6. Cross-Functional Teams
    1. Structuring Cross-Functional Teams
  7. Developing Engineering Processes
  8. Practical Advice: Depersonalize Decision Making
    1. Code Review
    2. The Outage Postmortem
    3. Architecture Review
  9. Assessing Your Own Experience
Toc
0 results found
Chapter 9. Bootstrapping Culture
2020/06/02

When you are in the role of senior engineering leader, part of your job is to set the culture of your function. A common failing of first-time CTOs is to underestimate the importance of being clear and thoughtful about the culture of the engineering team. Whether you are growing a new team or reforming an existing team, neglecting the team culture is a sure-fire way to make your job harder. As the team grows and evolves, it’s important to attend to your culture as you would attend to any other important piece of infrastructure that you rely on.

At Rent the Runway, I had the opportunity to set up many of the cultural elements of the engineering team. Because the team was still running on the classic, unstructured “scrappy startup” model when I joined, I was able to introduce many cultural structures and practices to both the team and its members. This process was a great learning experience for me.

For many people who are attracted to startup culture, the ideas of “structure” and “process” are seen as pointless at best and harmful at worst. I have seen surveys of startup teams in which the idea of introducing structure evoked such reactions as “slow” and “innovation-crushing.” These respondents believed structure is the reason large companies move slowly, foster bureaucracy, and are generally boring places for bright people to work.

When talking about structure with skeptics, I try to reframe the discussion. Instead of talking about structure, I talk about learning. Instead of talking about process, I talk about transparency. We don’t set up systems because structure and process have inherent value. We do it because we want to learn from our successes and our mistakes, and to share those successes and encode the lessons we learn from failures in a transparent way. This learning and sharing is how organizations become more stable and more scalable over time.

I’m hoping to help you develop a personal philosophy on company culture in addition to giving you ways to set up process and structure. If you want to create healthy teams, you need to have a sense of what is important to you, to your company, and to your growing group of colleagues. Consider not only what you care about, but also how you can scale that knowledge and effort effectively as the company and team grows and evolves. You’re going to be trying out structures and processes and learning from them, but it’s hard to learn if you don’t have a basic theory to test, and you don’t set out to prove or disprove hypotheses about that theory. So let’s approach this culture-creating exercise scientifically, and see how you can think about the pieces of culture you might need in a logical fashion.

Early startups attract people who are capable of dealing with extremely high amounts of uncertainty and risk in exchange for equally high degrees of freedom to operate. There is no long-term guarantee that the company will succeed or even continue to exist for very long, no matter how strong the idea seems on paper. Often the market is unproven. Some signs look good and other signs look bad. There may be fierce competition from other companies, big and small. Furthermore, there is very little established work to build on. The code is unwritten. The business rules are not set up. It is hard to overstate how many decisions need to be made in the context of a startup, even one that has been growing for a couple of years. Everything from deciding on technology frameworks to deciding on office decorations is up for grabs.

Many of these initial decisions will be undone a couple of times before they settle. It’s easy to think about changing a framework that didn’t scale well with the company’s technology needs, but things like vacation policy, core office hours, and even company values could change and evolve in a startup’s first few years.

The most important thing for leaders to be willing to do in those early days—and leaders generally includes everyone in the company, not just the founders or executives—is to pick a strategy and run with it. Cultivate decisiveness in the face of a massive number of options. You have a problem? Figure out a solution and fix it. That solution doesn’t work? Try something else. You don’t need to find the perfect solution; you need to find something that will get you through to the next milestone, whether that milestone is the next release, the next growth spurt, the next funding round, or the next hire.

Sometimes companies decide to limit the decisions themselves, as in an organization that foregoes titles. Having no titles is in one sense a decision, but in another sense it’s a decision that means you never need to decide what someone’s title will be, you won’t need to worry about promoting people to new titles, and you don’t need to build up the apparatus that will make future decisions about titles because you have removed that as an option. Deciding not to decide right now is a popular option for new companies, because it really doesn’t matter at the scale of a few people.

One of the greatest writings about organizational politics is a piece called “The Tyranny of Structurelessness” by Jo Freeman. While the article is about early feminist/anarchist collectives, Freeman’s insights apply equally well to startup culture. Pretending to lack structure tends to create hidden power structures resulting from the nature of human communication and the challenges of trying to scale that communication. Interestingly, Freeman describes a set of circumstances in which the unstructured group can, in fact, work:

  1. It is task oriented. Its function is very narrow and very specific, like putting on a conference or putting out a newspaper. It is the task that basically structures the group. The task determines what needs to be done and when it needs to be done. It provides a guide by which people can judge their actions and make plans for future activity.

  2. It is relatively small and homogeneous. Homogeneity is necessary to insure that participants have a “common language” for interaction. People from widely different backgrounds may provide richness to a consciousness-raising group where each can learn from the others’ experience, but too great a diversity among members of a task-oriented group means only that they continually misunderstand each other. Such diverse people interpret words and actions differently. They have different expectations about each other’s behavior and judge the results according to different criteria. If everyone knows everyone else well enough to understand the nuances, they can be accommodated. Usually, they only lead to confusion and endless hours spent straightening out conflicts no one ever thought would arise.

  3. There is a high degree of communication. Information must be passed on to everyone, opinions checked, work divided up, and participation assured in the relevant decisions. This is only possible if the group is small and people practically live together for the most crucial phases of the task. Needless to say, the number of interactions necessary to involve everybody increases geometrically with the number of participants. This inevitably limits group participants to about five, or excludes some from some of the decisions. Successful groups can be as large as 10 or 15, but only when they are in fact composed of several smaller subgroups which perform specific parts of the task, and whose members overlap with each other so that knowledge of what the different subgroups are doing can be passed around easily.

  4. There is a low degree of skill specialization. Not everyone has to be able to do everything, but everything must be able to be done by more than one person. Thus no one is indispensable. To a certain extent, people become interchangeable parts.

Here Freeman describes a common scenario for many early-stage startups. Even when the overall company grows beyond the small group, the engineering team often pushes itself to stay unstructured. Hiring “full stack” engineers who are exclusively sourced from the professional and social networks of the current team results in low skill specialization and high homogeneity. Forcing the team to be collocated lowers communication barriers. And perhaps most critically, having an engineering team that operates solely as the execution arm of the product or founder makes the team highly task-oriented.

I will hazard a guess that some folks may bristle at this characterization of the common startup technology organization. After all, these engineering teams are often the well-paid darlings of the company! Be that as it may, the unstructured organization either displays characteristics that ultimately make it less self-directed than the members might wish to believe, or is run by hidden hierarchies and power dynamics. In many cases both things are true to some extent.

The example of the structureless team also applies to technical decisions and processes. There is a reason that you often find a lot of spaghetti code in early startups. When work is done to satisfy an immediate task, in a unified code base worked on by a team of interchangeables, the result is not usually a larger thoughtful structure, but a tweak here, a hack there—anything to get things done and moving forward. It’s no surprise that we usually end up refactoring spaghetti code when we want to make it scalable, because refactoring usually involves identifying and explicitly drawing out structure in order to make the code base easier to read and work in.

That, in short, is the value of structure. Structure is how we scale, diversify, and take on more complex long-term tasks. We do it to our software, we do it to our teams, and we do it to our processes. In the same way that strong technical systems designers are capable of identifying and shaping underlying system structures, strong leaders are capable of identifying and shaping underlying team structures and dynamics, and doing so in a way that supports the long-term goals of the team and equips the individuals to achieve their best.

Nothing is more ridiculous than a small team with a rigid hierarchy. We would all think that a team of five people where the fifth reported to the fourth, who reported to the third, who reported to the second, who reported to the first was pretty strange and probably unnecessary. Similarly, if a team of five in a struggling business spent most of their time in meetings deciding which toilet paper to stock in the bathroom, their priorities would seem skewed. Structure can come too early, and cause harm by slowing down a group that should be focused on other things.

However, it’s more common in small companies to see structure come too late. The problems creep up slowly. One person gets used to making all of the decisions and changing his mind frequently. This strategy works fine when it’s just him and a couple of others. But when he keeps doing it with a team of 10, a team of 20, a team of 50, what you start to see is a high degree of confusion and wasted effort. The cost to change his mind becomes more and more expensive.

One of the best analogies I’ve heard for startup leadership comes from a friend, On Freud, who’s been in engineering management at several different startups. On describes the earliest startup as like driving a race car. You’re close to the ground, and you feel every move you make. You have control, you can turn quickly, you feel like things are moving fast. Of course, you’re also at risk of crashing at any moment, but you only take yourself down if you do. As you grow, you graduate to a commercial flight. You’re farther from the ground, and more people’s lives depend on you, so you need to consider your movements more carefully, but you still feel in control and can turn the plane relatively quickly. Finally, you graduate to a spaceship, where you can’t make quick moves and the course is set long in advance, but you’re capable of going very far and taking tons of people along for the ride.

Assessing Your Role

Recognize the size of the vessel you’re steering. This will be determined by a combination of the number of people in the company, the age of the company, the size of the existing business infrastructure (software, processes, and the like), and risk tolerance:

People

The more people you have, the more thoughtful structure you need to get everyone moving in the right direction. Leaders who want a high degree of control over their organization tend to need more structure in place to make sure their wishes are enacted. Modern companies often put their structural focus on goal setting instead of trying to make all decisions from the top, but don’t underestimate the structure you need to successfully set and communicate goals.

Age

The longer a company is around, the more habits become entrenched. On the other hand, the longer a company has been around, the more likely it is to continue to survive.

Size of existing infrastructure

If you have few established business rules (such as “this is how we determine what to charge our customers”) and little code or physical infrastructure (like stores, warehouses, or inventory), there is less need for structure. On the other hand, the more existing business rules and infrastructure you have, the more you’ll need clarity on how to handle them.

Risk tolerance

Are you in a highly regulated industry? Do you have a lot to lose if certain types of mistakes are made? Or are you in an unregulated industry, with little on the line? Your structures and processes should reflect this. In general, the more people you have depending on you and the larger the business is, the less risk you’ll be willing to take even without regulatory requirements.

Structure grows as the company grows and ages. In fact, there’s even a law that accounts for this, from John Gall’s book Systemantics[1]:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

Your company started as a very simple system that contained a few people, and as more and more people and rules and infrastructure were added, it evolved into a complex system. I don’t think there’s a huge benefit in overdesigning your team structure or process when your team is small and functioning well. However, at some point you’ll start to experience failure, and failure is the best place to investigate and identify where your structure needs to change. In the creating a career ladder example, one person quitting because of a lack of a career path might not be enough to push you to create a career ladder, but you may reconsider when multiple people quit or fail to join. You’ll need to weigh the value the lack of structure brings the team against the cost of losing people you might otherwise want to employ.

My advice to leaders is simple: when failures occur, examine all aspects of reality that are contributing to those failures. The patterns you see are opportunities to evolve your structure, either by creating more or different structure or removing it. Think about how often the failure happens and its cost, and use your best judgment about the changes that need to be made. Using failure to guide evolution lets you apply structure at the right level. If a failure is occurring in only one part of the system—say, on one team—you can try to address the structure on that team without necessarily changing the larger structure. What about examining success? Well, you can learn things from success, but it is often a poor teacher. Ironically, while luck plays a role in both failure and success, we often attribute failure to bad luck and success to our own actions. As Gall’s law says, a simple system that works can evolve into a complex system, but that doesn’t mean that applying the lessons from a successful complex system will let you replicate that success in other places. As humans, we tend to blame failure on bad luck until it’s impossible to ignore our own contributions to that failure. Therefore, we’re less likely to overstructure our teams based on lessons from failure. Success, on the other hand, tempts us with the silver bullet, that one weird trick that could make everything great. If you want to learn from success, make sure you can identify the actual improvement you’re seeking when applying those lessons more broadly, and that you understand the context required to repeat that success.

The age of the company and size of the team plays into this issue. If you’re at a company that’s been around for a while and will be around for a while, using structure (adding or removing) to improve efficiency is very helpful, even if it costs something up front to implement. That’s part of the trick. Learning rarely comes for free. Analyzing situations and thinking about good takeaways takes time. If the value of your future time is less than the value of your current time, then you’re probably not going to worry too much about saving future time. Just because your company is big, old, and stable doesn’t mean you can have as much rigid, unchanging structure as you want. Technology changes often enable formerly risky moves to become safer than the slow-moving alternatives. Software release frequency is a good example of this. For a long time, releasing software frequently was difficult and expensive, largely because you were shipping that software to the user. In the modern SaaS world, bugs can be easily fixed, and the risk involved in shipping a bug is much lower than that of not expanding features quickly enough to keep up with competition. It’s this type of unconditional attachment to old structures that makes many people hesitant to adopt structure at all. But if you don’t adopt structure when you need it, things can also go wrong.

When every new hire slows the team down for months because there is no onboarding process, that is a failure due to lack of structure. When people regularly leave the company because they have no path to advancement or career growth, that is a failure due to lack of structure. The third time you have a production outage because someone logged directly into the database and accidentally dropped a critical table, that is a failure due to lack of structure. I said earlier that I prefer to talk about learning and transparency rather than using the word structure, because really what we’re talking about here is identifying the causes of failures, especially frequent failures, and trying to figure out what we can change to solve for those failures. This is fundamentally about learning.

Creating Your Culture

Culture is how things get done, without people having to think about it.

Frederick Laloux, Reinventing Organizations: A Guide to Creating Organizations

Inspired by the Next Stage of Human Consciousness

Culture is an oft-discussed topic of building startups. What are the core values of the company? What is the company culture like? Are new hires “culture fits”? Is “culture fit” a dogwhistle for discriminatory hiring practices?

One of the things I have come to believe strongly is that culture is real; it’s also incredibly important, and it’s something that many people don’t understand at all. It’s both an easy, natural consequence of your company’s evolution and something that can quickly become a problem if you don’t tend to it. Consciously guiding the culture of your team is part of a leader’s job, and to do this well, you need to understand what it means in the first place.

So what is culture? Culture is the generally unspoken shared rules of a community. American culture dictates that we shake hands as a greeting, for example, while in some other cultures, touching strangers is considered very odd. The way you address people of different standings or different relationships to you is part of your culture. Culture doesn’t mean that every single person holds exactly the same values, but it tends to guide a general overlap, and it creates a bunch of rules of interaction that you don’t have to think much about if you are deeply ingrained in that culture.

People do make decisions using methods other than cultural values. They may adhere to the standards of a formal or informal contract, for example. They may do a pure data-driven analysis and determine the optimal outcome. But in complex environments where the needs of the group must override the needs of the individual, cultural values are the glue that enables us to work as a team and make decisions when faced with uncertainty. This is why figuring out and guiding your culture is such an important part of building a successful company.

If you’re forming a new company, there’s no guarantee that a predetermined healthy culture will fall out. You may hope that you can create a planned community of people, a community of like-minded individuals who will bind together to create this great workplace and product. But reality is much messier than that. Reality is much more of a race for survival, with culture as an afterthought or a post hoc justification. The early employees will form the culture, for good or for bad—or likely for a mixture of both.

Not every person will fit in at every company. The sooner you realize this, the better. Sometimes we are afraid to have core values because we believe they will create discrimination. I would argue that a thoughtfully created set of values that are actually values should reduce the kinds of surface discrimination that often happen at tech companies in favor of creating a real community of employees who share core principles and ways of communication. It is to your advantage to create a culture that allows for bringing a broader range of people into your community. “Engineers who graduated from MIT” is not a culture. “People who value technology innovation, hard work, intellect, scientific process, and data” might be. The first allows only an incredibly narrow subset of humanity to pass through it successfully. The second allows a much broader set of people to fit, while ensuring those people actually have the same values.

If you come into a company with core values, those values were probably created by the founders, or founders and early employees, and thus they reflect the company’s culture. This is important to understand, because you’ll be measured against these values whether you realize it or not. The founding team’s values will be reinforced, recognized, and rewarded inside of the company. My experience has shown that employees who truly embrace and exhibit all of the core values of a company tend to do well naturally. The fit is easy for them. They may get stressed out or work too hard, but they are well liked and usually happy. Those who do not match all of these values as easily will have a harder time. That doesn’t mean they will fail, but there will be more friction for them, and it may feel like more work to fit in and feel accepted.

How does this apply to you? If you are a technical executive, cofounder, or CTO, this information has deep applications. If you join or create a company with very different values than your own, you’ll feel a great deal of friction that will make your life harder. At the highest levels, all of this cultural alignment comes to play in everything you do, because you spend so much of your time in the land of negotiation, collaboration, and cross-functional teamwork. This doesn’t mean that you can’t be successful in a company that holds some different values from your own. In fact, it’s pretty rare that you agree perfectly with every value of every person on the senior team of a company. You probably don’t even agree with every value of every person in your family, or among your friends! Still, the amount of overlap between the traits you value most and the traits your company values most largely determines how easy the fit will be for you.

Applying Core Values

Whether or not you’re in a founding or executive position, understanding and cultivating culture is a key part of your job as a leader. Here are some suggestions for how to approach this issue.

First, define your culture. If you have a set of company values, map those values onto your team. You may add a couple of values that are special to your team, or interpret the values in a way that makes sense for your team. On my tech team at Rent the Runway, for example, we explicitly valued diversity. That meant that we were more interested in what you could do and what your potential was than having you fit a certain set of checkboxes in the screening process. We layered a learning culture on top of our company values, because we believed that this was important for us as engineers. The point of this layering is that every subteam will have its own slightly distinct culture. Some teams are focused on being very professional, are in the office for very regular hours, and work in a very regimented way. Some teams prefer later or earlier hours, or less formal meeting cultures, with more room for chatting and hanging out socially.

Second, reinforce your culture by rewarding people for exhibiting its values in positive ways. People can share core value stories at company all-hands meetings. At our technology department all-hands meetings, we would have people give shoutouts to each other for “keeping it dope” and going above and beyond. Some people find this exercise uncomfortable, myself included. Reach through the part of you that is shy about praising people or embarrassed to share your feelings, and go into the part of you that cares about the people you work with. You can share these stories in a way that is not forced or fake. The stories that we tell as a community bond us together.

One of the most important uses of performance reviews is to evaluate the alignment between team members’ values and the company’s values, and therefore what values should be part of your performance review process. Call out when and how people exhibit some of the core values of the team. This practice reinforces desired behavior in a positive way. It also gives you a sense of who on your team exhibits most or all of the values, and who does not.

Learn to spot people who have values conflicts with the company or team. If your company has a value of “roll up your sleeves and get involved,” the teammate who continually pushes off work to others is not truly following this value of the company. If you have a value of “happiness and positivity is a choice,” the teammate who pooh-poohs every idea and criticizes everything is going to have problems fitting in. Sometimes, people will change to adopt the values. “Happiness and positivity is a choice” is actually one of the core values of Rent the Runway, and I would not say that I came from a work background of happiness. In fact, I came from a fairly professional and critical work culture. But I learned to appreciate the value of looking at things in a positive light. That doesn’t mean that I lost my critical eye, and it was never the easiest value for me to adopt completely, but it wasn’t a deal breaker. Using the core values to coach people in areas where they are misaligned can help you articulate what otherwise may feel like just ambiguous friction.

Finally, use this as part of your interview process. Remind your interviewers of the values of the team, and ask them to look out explicitly for places where the interviewee seems to match or collide with these values. A lot of interviews try to determine cultural fit by what I would call “friendship” markers, such as “Would you like being stuck in an airport with this person?” You certainly don’t want to hire people that your team can’t stand to be around, but cultural fit is not about hiring friends. I’ve had great working relationships with people that I would not want to chat with for hours outside of work, and terrible working relationships with people I would love to be stuck with in an airport. Furthermore, culture fit as determined by friendship tests is almost certain to be discriminatory in some way. Humans form friendships with people who have significant shared background experiences, and these experiences tend to closely correlate with things like schooling, race, class, and gender. The shortcuts you get by hiring friends are not usually the values you need to form a strong team.

So, don’t be vague when discussing fit. Be specific. What are the values of this team, and where have you noticed any match or mismatch? A very smart engineer who really values independence may not be a fit for a team where everyone must collaborate extensively on all projects. Someone who believes that the most analytical argument always wins may not work well in a company that values empathy and intuition over pure analytical skill. I use these examples because all of the values here are compatible in certain situations, and incompatible in others, and that is what makes this a powerful measure. Understand what your company’s values are, understand what your team’s values are, and think about what you personally value. Write the values down if they aren’t already written, and try to be explicit. Use this explicit list to evaluate candidates, praise team members, and inform your performance review process.

Creating Cultural Policy

Creating cultural policy documents can be hard, because getting started on these documents from scratch is hard. Fortunately there are fewer and fewer documents that you need to start from scratch to create, as more people are sharing publicly their policies and processes for everything from career paths to pay scales to incident management. However, just having a starting point and copying it is not always enough. I learned this the hard way when I tried to roll out my first engineering career ladder. As I said earlier in this chapter, there comes a time for adding structure, and that time is usually when things are failing. The failure that drove me to create a career ladder came when our HR team was doing a salary review for the engineering team. I realized that we had no salary structure at all. Because of that lack of structure, most people were paid based on a combination of their previous jobs’ salary and their negotiating skills. Additionally, we had a hard time figuring out who we needed to be hiring in. Were we only hiring “senior” engineers? What did that mean? What about management or other roles?

After a nudge from our HR team, I set out to create a ladder, which I’ve cited in pieces throughout the book. I did this by asking my friends who ran other startups if they had one. One of my friends did, and he shared it with me. It had eight levels, from entry-level engineer to executive, broken into four categories: technical skills, getting stuff done, impact, and communication and leadership. I took this ladder, added a few more details, renamed the levels, and rolled it out. This makeshift ladder was very basic. For each level, at each skill, you got one or maybe two sentences on what classified a person as working at that level. Even with some additional information from me, there were perhaps four points you could look at for each category. The worst were the earliest levels, which were the most basic and provided very little guidance to early-career engineers. I delivered the new ladder to my team, and even communicated the new ladder in the same style that my friend used to communicate it to his team. I told them the ladder existed to make sure we were being fair with things like compensation, and it was something they could use to discuss their level with their manager and learn how to grow. I told people it wasn’t a big deal, that they shouldn’t obsess over their level. I then spent some time talking about John Allspaw’s blog post “On Being a Senior Engineer” in an attempt to inspire the team to push themselves.

Long story short, my first ladder was a flop.

Why did a ladder that seemed to work fine for my friend fail so badly for me? I can only speculate, but there were some pretty big differences between our companies. My company was very diverse in terms of background. I had a team that was mostly pulled from small companies and startups, with a handful of people like myself who had worked at big finance companies and only a couple who had mostly worked for big tech companies. We had no real shared cultural habits to pull from because of this diverse set of work experiences. My friend, on the other hand, managed a team that had a very large, strong core of people who had all worked for the same large tech company, so there was a lot more shared understanding that didn’t need to be made so explicit.

I share this story for a very important reason: where my friend was able to succeed, I failed, despite following the same template. This lesson is crucial for anyone who wants to create good team culture. What works for one company—a company that is creating a certain type of product or working in a certain industry—will not always translate well to another company, even if the companies have a lot of things in common. We were both managing startups when we rolled out our respective ladders, and our teams were similar in size, but we needed very different things for our teams to be successful. My first ladder was a flop because my team needed more details. The goal of the lightweight ladder was to keep the team from obsessing over their levels and promotion, but instead the lack of detail caused many of them to obsess even more. Engineers argued that they deserved to be at higher levels because the details were vague. It caused a constant series of headaches.

Writing a Career Ladder

Here are some important issues to consider when writing a career ladder for your organization:

  • Solicit participation from your team. To write a better ladder, I had to change my approach. First, instead of doing it by myself, I enlisted the support of the senior managers and engineers on the team to provide feedback and details. I asked people to highlight things they didn’t understand. I asked them to propose rewrites, additions, edits, and details. We discussed it as a group, and we had subgroups work on the parts of the ladder that they cared about most. For example, the most senior individual contributors worked on the technical and skills expectations for the individual contributor levels.

  • Look for examples. Second, I got more examples of ladders from friends at other companies to help provide some ideas for the details. There’s a lot of good work out there now that you can use if you need to write something, but at the time, I had to do my research by asking people to print out whatever they felt they could share, or give me high-level notes. The best details came from friends at bigger employers, especially those with strong technical reputations. It can be hard to explain the scope of work expected at very senior technical levels, and having those examples from bigger companies really helped us put the details in writing.

  • Be detailed. One of the biggest challenges you’ll face when writing a good ladder is sketching out the details. You want something that is inspirational and descriptive but that matches your company. It doesn’t make sense to expect a director for a 50-person engineering team at a startup to manage, say, an entire division, in the same way that it might make sense to have that expectation at a large multinational corporation. Think about the kinds of details you would look for when deciding if someone should be hired in at a level or promoted to a level, and try to include those details as appropriate.

  • Use both long-form descriptions and summaries. I broke the ladder out into two documents. The first was a shorthand spreadsheet version that allowed me to see the various level attributes side-by-side and see how they evolved through increasing levels. This was helpful because as I wrote, I could see how I built from one level to the next, and how the roles expanded in their scope, skills, and responsibilities. The second document was the long-form version. Writing the long-form version was helpful to me because I felt that I could tell a more complete story about the players at each level. Instead of just visualizing the level as a set of skills and attributes, the long-form ladder reads a bit like a performance review of a person operating well at each level. You—and your employees—can see how those skills work together to form a complete role. How many levels should your ladder have? You’ll need to answer two more questions to figure this one out. First, how do you pay people? And second, how do you recognize achievement?

  • Consider how the ladder relates to salary. Your HR department will want to use the career ladder to help set salary expectations. Usually, each level will have a salary band, or a range between a minimum and maximum base salary that a person in the level can earn. If you don’t have many levels, you’ll need to have very wide salary bands to account for the fact that two people within that level can perform very differently, and to account for the fact that engineers tend to expect to get salary increases frequently, especially in the earlier parts of their careers.

  • Provide many early opportunities for advancement. Some people advise having a lot of levels toward the beginning of the ladder to account for the fact that early-career engineers expect frequent raises and promotions. You may want to be able to promote someone every year for the first two to three years of her career. If that’s the case, create several levels that encompass the role of software engineer and provide relatively narrow salary bands for those levels, on the expectation that people in those roles are either being promoted quickly or moving on from your company.

  • Use narrow salary bands for early-career stages. Lots of levels and narrow salary bands mean that you can promote people quickly and justify giving them raises while keeping your pay for all people at a certain level close to the same. This is good if you are worried about paying fairly and avoiding bias that might lead you to, say, pay men more than women at the same level. Unfortunately, it’s incredibly hard to create enough detail between close levels to allow a person to easily distinguish someone being at one level or another.

  • Use wide salary bands when and where you have fewer levels. Wide salary bands and few levels make a clearer distinction between the skills at each level, and should make it easier to tell who is operating at which level. In the case of widely spaced levels, you want to have large salary bands and you want those salary bands to overlap. So, a software engineer band may go $50–100K, and a senior software engineer band may go $80–150K. That means a strong software engineer may make more than a senior engineer. You need this wiggle room to retain talent who are performing well at their current levels but are not ready to take on the additional responsibility of the next level. You will also find yourself using this wiggle room to hire people who are on the fence into the lower level with the expectation that they will be promoted quickly.

  • Consider your breakpoint levels. It is common for companies to have certain levels that they consider “up or out.” These are early-career roles where a lack of advancement means that the person has not achieved the maturity or independence needed to remain at the company. This policy tends to get translated into your ladder as an implicit or explicit breakpoint level. What is the lowest level at which people can sit forever, never getting promoted but also not underperforming? This is your breakpoint level. For many companies, it’s somewhere around senior engineer. Someone who’s made it this far is a solid team member, but he may stay at this level indefinitely by his own choice. It’s good to have a notion of where this is. You may even want to use it as the point at which your ladder levels get harder to achieve. Expect your team to cluster around this level, with fewer people above or below it.

  • Recognize achievement. Some companies want to keep levels secret, but that tends to be impossible. People will talk. However, you can go out of your way to emphasize certain levels while keeping other levels secret, possibly even from the employees themselves. Some HR departments have separate pay grade numbers that they use to track employee pay that are disconnected from career ladders entirely. I am not advocating for this. However, I do encourage you to have at least some of your levels be keystone promotions, which are shared and celebrated. I think that the promotion to senior engineer is a big deal, as well as the promotion to staff engineer and, if you have such a role, principal engineer. On the management track, a promotion to director is worth celebrating, as is a promotion to VP. Having keystone levels that are not too close together gives people a bigger achievement to strive for beyond the next pay increase, and keeps these levels feeling important from a larger career standpoint.

  • Split management and technical tracks. It’s pretty obvious in this day and age that you need separate tracks for management and individual contribution. You do not want people to feel that the only path to advancement is by managing people, because not everyone is suited to that role. Commonly, you’ll see a split above senior engineer where organizations start to specify management levels and technical levels. However, you should not necessarily expect to have the same number of people in the senior technical levels as you do in the senior management levels. Senior management is generally a volume-driven need. You need enough managers to manage the people you have on the team. Senior technical depends on the complexity and scope of technical leadership that your teams and products require. It is possible to have a large team with few senior technical people, or a small team with many senior technical people and fewer managers. It would be unusual to have a perfect balance here.

  • Consider making people management skills a mid-career requirement. Encourage everyone to have some sort of management or mentorship experience before they are eligible to be promoted above the level of the track split. For most companies, the tracks should split when people start to exhibit leadership, whether that leadership involves managing humans or designing software. But even when designing software, you’re dealing with other humans and human needs. Great senior individual contributors still know how to manage projects and mentor more junior members of their team, so consider making leadership experience (usually via acting as a tech lead) a requirement for promotion to senior individual contributor levels.

  • Years of experience. No one likes to put artificial barriers onto people, and years of experience can feel like the most artificial barrier. With that said, I encourage you to be wise on this issue. In my ladders, I distinguish the keystone levels by an expectation of maturity increases, and these tend to correspond with years of experience in the industry as well as, to a lesser extent, age. For example, take the case of staff engineer. It takes a lot of individual maturity to think through large projects, which is, in my view, the distinguishing feature of a staff engineer. Being a brilliant programmer is not enough to be a great staff engineer; you need to have shown a track record of completing and supporting some long-running work to justify this title. You don’t have to put years of experience as a strict requirement for levels, but consider having some rules of thumb, especially if you are writing a ladder for the first time and rolling out levels.

  • Don’t be afraid to evolve over time. When you write a ladder like this, you’re creating a living document that will need to evolve as your company grows. You’re probably going to miss some details. My ladder was hard for frontend-focused developers to interpret because of my own focus on infrastructure development, so we needed to tweak it to better account for what it meant to be a senior performer within that world.

A good ladder is a critical element to use in hiring, in writing performance reviews, and of course in the promotion process. If you have the chance to create such a document, don’t be afraid to involve your team. The best processes and documents reflect the team as a whole and not just your bias at the moment, and one of the greatest things about setting these ladders up in a small company is that you can involve a lot of people without a ton of bureaucracy in the process.

Cross-Functional Teams

Who do you work with? Who do you report to? Who do you collaborate with? The answers are obvious in both extremely small companies (answer: everyone) and very large companies (answer: there’s a pretty clear structure that was set up before you joined). As a leader at a growing company, you’ll need to help answer these questions for your team and your company at least once, and probably multiple times. What should the answers be?

I want to take some time to talk about one of the best things I experienced in my work at Rent the Runway: the evolution of our product engineering organization. When I joined, the engineering team was divided into roughly two groups: storefront, which did all development for the customer-facing website, and warehouse, which supported the software that ran the warehouse operations. We quickly evolved storefront to be frontend and backend because we were rewriting the code from a PHP monolith to a Java- and Ruby-based microservices architecture.

Toward the end of my first year, we ran an experiment. We had a new product we wanted to build for the customer, a feature based on our customer photo reviews. Because finding a dress that would fit well was a challenge for our customers, we wanted to enable shoppers to see photos that other customers had uploaded showing themselves in the dresses, along with customer-provided information about their normal size, height, weight, and “shape” (athletic, pear, curvy, etc.). To implement this feature, we created a cross-functional team. We had engineers who specialized in the frontend user experience development, and engineers who worked on the backend services. We had a product manager, designers, a data analyst, and even a representative from the customer service team. This cross-functional team worked as a group to design and deliver this feature to our customers.

This project was a massive success. We delivered a good feature, fairly quickly, and the contributors all felt that they understood the goals of the project and were able to work better because of this cross-functional team. Prior to this project we had been deep in a pattern of “us versus them,” where your particular business function was “us” (tech, product, analytics, marketing, etc.), and the rest of the organization was “them.” Creating these collaboration units gave people a chance to see the whole group as “us.” It was a clear win in terms of organizational health, so we evolved our whole organization to have all product engineering performed by cross-functional teams like this. Call them what you want—“pods” or “squads” or “pillars”—but cross-functional product development groups are a popular structure for a good reason. By putting everyone who is needed to make a project successful together in one group, you help the members of those teams focus on the project at hand, and you make the communication for the whole group much more effective.

Conway’s Law is often cited in discussions of this kind of structure. It states: “Organizations which design systems…are constrained to produce designs which are copies of the communication structures of these organizations.”

When we put cross-functional teams together, we are acknowledging that the most important communication—the communication that we need to favor above all else—is that which leads to effective product development and iteration. Note that this structure will not necessarily produce the most effective technology! In fact, it will probably produce systems that have some inefficiencies compared to companies that have a more engineering-centered team structure. So, should you adopt this structure, you have to decide where you’re willing to take some system design hits in order to most effectively create products.

Structuring Cross-Functional Teams

How do the nuts and bolts of such “pod” structures work? One element that often causes anxiety is who is managing whom. When we moved to this team organization, we did not change the management structures. Engineers were managed by engineering managers and reported in to me. Product managers reported to the head of product. But determination of who was working on what was done largely by the pod itself. This meant that you could still get technical guidance and oversight from your engineering manager, but your day-to-day work was determined by the needs of the pod’s roadmap.

Of course, every function has its own focused needs. Usually someone in engineering needs to oversee critical core systems, and you probably need a few specialists around for things like the core web platform, mobile, or data engineering. I kept these functions in a small infrastructure organization that was not generally assigned to product development. Even with a dedicated infrastructure group, the engineers assigned to product pods still need some time to account for engineering-specific tasks like on-call, interviewing, and sustaining engineering (aka technical debt). I advise reserving 20% of all engineering time for such work, based purely on my personal experience and the experience of my peers in engineering management.

This cross-functional structure is not unique to small startups. Many large companies also structure their teams in this fashion. Banks, for example, often have technology teams that are attached to specific areas of the business, and while the management structure is formed by engineers, the roadmap and day-to-day work are jointly determined by the needs of the business unit and its associated engineering team. There is generally a centralized infrastructure team that supports both fundamental systems as well as large frameworks and technologies that will be used by many teams across the company. Even many technology companies are structured in this way, although the “business units” may themselves be headed by former engineers who act as product or business managers instead of business specialists.

The implications of the cross-functional structure are subtle. The values of everyone in these teams will start to change. In technology-focused structures where engineers work solely with other engineers, particularly engineers of their same “type” (mobile, backend, middleware, etc.), the focus is on being the best engineer by some measure of engineering excellence. People who design complex systems or who know the details of the latest iOS are the leaders and role models for the teams. In a product-focused structure, the leadership focus changes. Now the engineers who have the best product sense, the engineers who are capable of getting features done quickly and efficiently, and the engineers who communicate the best with the other functions will start to emerge as the leaders of the team.

I mean no value judgment here, but I encourage you to be aware of the product/business versus technology focus and apply it where it makes sense. What is truly important to the success of your company or your organization? If the most important thing is evolving a product that is a function of many different business areas coming together, you probably want leaders who have that business sense. On the other hand, in the areas where the technology must be rock-solid or exceptionally innovative and cutting-edge, you probably want teams that have more of an engineering focus and that are led by people who can design complex systems. You don’t have to go entirely one way or the other, but recognize that one of these will lead the company as a whole, and—especially if your role is in senior management—focus your skill set on the one that the company itself most values and hire in for the other.

Developing Engineering Processes

I’ve had to deal with many different engineering processes over the years. I remember the first time I worked in a code base with unit tests that we were expected to run before checking in code. I was very diligent about doing this, and very upset every time someone broke the build because she hadn’t bothered to make sure her change didn’t break the tests. I also remember the first time I had an engineering process forced upon me that I hated. After years of no required code reviews, no ticketing, and no tracking, a central bureaucracy decided that everyone had to adopt all of these measures at once in a push for standard software development lifecycle management. It felt unnecessary, slow, and burdensome, and no one bothered to explain to us why these changes were happening.

Engineering processes are the place where the rubber meets the road when it comes to structure. Career ladders, values, team structures—all of those are easy compared to the general angst and frustration that you can cause by adopting the wrong engineering processes for your teams. Without any process, your teams will struggle to scale. With the wrong process, they will be slowed down. Balancing the current size and risk tolerance of your team with the processes at hand is the essence of guiding good software development and operational guidelines.

ASK THE CTO: ENGINEERING PROCESS

I’m the head of engineering at a small but quickly growing startup. We have very little process right now: there are no code reviews, we use Trello to manage tasks but rarely put everything into that system, and our architecture decisions tend to be made by whomever is working on the project at the time, with my sign-off.

Recently, some engineers have come to me to complain that new people are checking bad code into the systems. They want us to introduce code reviews for all changes. I also just discovered that someone has been writing a new system in Scala, despite the fact that all of the rest of our code is in Ruby. He’s the only person who knows Scala on the team, and I’m afraid of the support burden, but the project is pretty far along so I can’t just kill it.

What should I do? I’m nervous about going from zero process to a bunch of process all at once, but something needs to change!

Think of process as risk management.

As your teams and systems grow, it’s almost impossible for any one person to keep the systems in her head. Because we have a bunch of people coordinating work, we evolve processes around that work coordination in order to make risks obvious.

One way to think about engineering processes is that they serve as a proxy for how hard or rare it needs to be for something to happen. A complicated process should exist only for activities that you expect to be rare, or activities where the risks are not obvious to people. “Complicated” in this case does not only mean a long process. Sometimes, the complexity is in getting sign-off from a group of people who are very busy, or in meeting a very high standard.

This has two important implications. The first is that you should not put a complicated process on any activity where you want people to move quickly and where you believe the risk for change in that activity is low or that the risks themselves are obvious to the whole team. If you want to do code review for all changes, make sure that the process for code review is not so onerous that the team slows down significantly on minor changes, because that will impact your whole group’s productivity.

The second implication is that you need to be on the lookout for places where there is hidden risk, and draw those hidden risks out into the open. There’s a saying in politics that “a good political idea is one that works well in half-baked form,” and the same goes for engineering processes. The processes should have value even when they are not followed perfectly, and that value should largely lie in the act of socializing change or risk to the team as a whole.

Practical Advice: Depersonalize Decision Making

There are three major processes that you should consider adding as your team grows. All of these processes work best when you set behavioral expectations around them in addition to the technical details.

Code Review

Code review is, for better or worse, a modern standard. Once you have a team of a certain size with a certain number of people working on a code base, code review can be a valuable tool for ensuring the stability and long-term quality of that code base. However, required code review will also be on the critical path of getting work done, so you want the process to be straightforward and efficient. Additionally, code review is often a place where engineers behave poorly toward one another, using it as a platform to criticize their colleagues or to enforce unrealistic standards. Here are a few best practices to smooth the way:

  • Be clear about code review expectations. For the most part, code reviews don’t catch bugs; tests catch bugs. The exceptions to this rule are that code review can catch missing updates to comments or documentation or missing changes to related features, and code reviewers can sometimes tell when there is inadequate testing of the new or changed code. Code review is largely a socialization exercise, so that multiple team members have seen and are aware of the changed code.

  • Use a linter for style issues. Engineers can waste absurd amounts of time on questions of style, specifically formatting. This should not be up for debate in code review. Decide on a style, and put that style into a linter that formats the code automatically. Allowing style to be up for discussion in code review often leads to nitpicking and criticism that can feel unproductive at best, and bullying at worst.

  • Keep an eye on the review backlog. Some companies implement a limit on how many outstanding review requests a person can have assigned to him, and they block that person from requesting review when he has too many outstanding requests. Think about how you want to get these requests pushed through the system, and how you can make sure that everyone gets adequate time in their code.

The Outage Postmortem

I’m not going to talk about the details of incident management, but the “postmortem” process is a critical element of good engineering. In fact, instead of calling the process a postmortem, many have started calling it a “learning review” to indicate that its purpose is not determining cause of death but learning from the incident. There has been a lot written on this topic, so I’ll highlight only a few elements that I believe are critical, especially for small teams:

  • Resist the urge to point fingers and blame. It is incredibly tempting, after a stressful outage, to point fingers and ask people why they failed to foresee the consequences of their behavior. Why did they run that command on that box? Why didn’t they test that? Why did they ignore that alert? Unfortunately, this blaming only results in people being afraid to make mistakes.

  • Look at the circumstances around the incident and understand the context of the events. You want to understand and identify the factors that contributed to this incident. This might include looking for tests that would have detected the problem, or tools that could have made the incident management go more smoothly. Getting a good list of these circumstantial contributors helps you detect patterns or areas for improvement, and forms the “learning” part of the learning review.

  • Be realistic about which takeaways are important and which are worth dropping. Be careful not to give the impression that people need to solve every problem they identify in the course of the exercise. Many learning reviews end in a laundry list of things that could be improved—everything from cleaning up alerts to adding role restrictions to following up with a third-party vendor to understand its API. It’s unlikely you will get to all of these, and in fact, it’s likely that if you try to do all of them, you will end up doing none of them. Pick the one or two that are truly high-risk and highly likely to cause future problems, and acknowledge the ones that you are going to let go for now.

Architecture Review

I’m going to roll into architecture review all major systems and tools changes that the team may wish to make. The goal of architecture review is to help socialize big changes to the appropriate group, and to make the risks for those changes clear. Some questions that you may ask people to come prepared to answer include:

  • How many people on the team are comfortable using this new system/writing this new language?

  • Do we have production standards in place for this new thing?

  • What is the process for rolling this out and training people to use it?

  • Are there new operational considerations for using this?

And here are some guidelines:

  • Be specific about the kinds of changes that need architecture review. Usually these include new languages, new frameworks, new storage systems, and new developer tooling. People often want to have architecture review to prevent teams from designing new features poorly, but it is usually unrealistic to try to catch new feature design early enough in a small company, and it’s hard even in a large company. It also slows things down a lot, and as to our earlier point, you probably don’t want to put a heavy process in front of a common activity like feature design.

  • The value of architecture review is in preparing for the review. Asking for review of big changes or additions to the systems forces people to think about why they want to make these changes. Again, one value of these processes is to help make people aware of risks that they may not have considered. You may or may not choose to have the team answer the question of why you should make this change in the first place. I have found that when people are willing and able to get through the requirements for making the change at all, the why is obvious.

  • Choose the review board wisely. You want the review board to include the people who will be most affected by the change, not just a static chosen group of gurus. Part of the goal is to get yourself out of the hot seat for making every technical decision, and part of the goal is to make sure that those who will need to deal with the outcome of the decision are part of evaluating it. You want these decisions to consider the wider team, and for the wider team to be bought in on them. There is no reason this needs to be company-wide. The scope of the deciding group is best kept to the people who will be closely impacted by the decision. There’s nothing more demoralizing than having someone from a completely unrelated area veto a project.

Assessing Your Own Experience

  • What policies do you have now? What practices? Have you written any of them down yet? When was the last time you revisited them?

  • Do you have company values? What are they? How do you recognize them in your team?

  • Do you have a career ladder? Do you feel it accurately reflects the team today? Does it reflect the team you want to have in the future? If not, can you improve it?

  • What risks are most concerning for your team? For your company? How can you mitigate those risks without burdening your team with unnecessary processes and bureaucracy?

Next Chapter: Conclusion


  1. 1 John Gall, Systemantics: How Systems Really Work and Especially How They Fail (New York: Quadrangle/The New York Times Book Co, 1975). ↩︎

打赏
支付宝
微信
本文作者:CodingRabbit
版权声明:本文首发于CodingRabbit的博客,转载请注明出处!