It was fun to write. I am not sure what impact it had on my intended audience back then, but I’ll try my luck again with this audience, with some mods on specifics without losing intent.

I’ll go back to the same FooStore from that earlier post (and yes, I really wish I had come up with something more creative than foo), and the same Janet who loves to buy foo.

Here’s the twist: foos are getting popular, and the store keeps running out of them. No amount of eventual consistency is going to help with foo inventories. Sure, a notification to users when new foo stock arrives is nice, but that ain’t cutting it.

So, Butterfield and others from foo.ai plan a new feature: Autofoo. A pre-order that’s paid for, and automatically fulfilled, on FIFO basis, as new stock comes in.

Here’s a standard-issue user story / epic for Autofoo:

This is all useful and necessary to be sure. But I want my team to care, deeply, about Janet’s emotions. Here’s how I would prefer to tell the user story:

To wit: Start with the user.

This is inordinately ornate to be sure, but the point was to make a point. It is also a trivial use case - the original one I wrote it for was significantly more complex. Furthermore, this is not intended to replace traditional epics and other artifacts. Rather, the hope is that every new feature or product begins life like this: with the user at the center of things.

As a perk: it’s a blast to write stuff like this (even if not as over-dramatized as this one), plus it gets the builders excited to build it.

I really do think screenwriters would make great product people!

Is it really as self-destructive as it sounds? I think it is at some level the opposite.

Here’s the thing: the early-stage CTO is a bit of an odd beast. I am speaking for myself here for the most part but have at least part-validation from some other similar oddballs that are close friends.

At least 3 things are true for us:

• We like building products, especially from scratch

• We don’t like building decks

• We like early-stage equity

Add those up, and the KPI really translates to protecting your investment: of time, blood, sweat, neurons, and other minerals involved in 0-to-1 builds. It aligns your success with that of the business.

In geekspeak, it’s the human version of all things we hold dear in system design: redundancy (it says it, right there!), transparent failover, single points of failure, resiliency.

I had a great chat recently with Joel Beasley on the Modern CTO podcast about exactly this — hopefully you find it entertaining, and perhaps useful!

You can also catch it on Spotify and Apple Podcasts (among others).

TL;DR: Don’t be a single point of failure!

“Can you write me the pseudocode for a depth-first search?”

This was a question posed to me in February 2003 by an insolent twerp of a coder at a now-defunct outfit called Random Walk Computing back in San Francisco. I’d been laid off twice in a span of 4 months. This was smack dab in the ember-shower of the dotcom inferno that had rendered the city, especially its SoMa neighborhood, a glass and concrete wasteland rather resembling a corporate version of post-Aegon Harrenhal. RWC was one of those “what have I got to lose” interview calls I took, preparing as I was to leave the city for good. (I didn’t for another 3 years, but that’s a different story.)

Anyway, straightforward as this piece of code is, I froze for a bit, trying to rehydrate dim CS 410 memories from almost a decade earlier (the last time I had to code up a fundamental algorithm). All the while, Insolent Twerp kept peering at me with his weird green glasses and highly unprofessional schadenfreude. I came this close to responding to him with “Only if you write me the pseudocode for Dijkstra’s Algorithm while I do it, you bombastic asshat”.

I got the code, didn’t get the gig. Idiots. Good thing too, as the firm was devoured by Accenture a few years later.

That’s when I decided that if I were to ever own the engineering hiring process at any place, I would never ever ask an interview candidate live coding questions.

See, I absolutely suck at live coding questions. I abhor them with a crimson passion. No doubt in large part because I suck at them, but more importantly because I believe that as a screening device, they are patently useless. They screen for the wrong skill. “Thinking on your feet”, while impressive, is not an imperative for a good software engineer. What are we trying to do here - write good software or defuse a bomb?

Besides, this approach racks up an impressive count of false negatives, ultimately making it a losing proposition for the hiring company. To add to that, it fails to give the candidate a good flavor of what it’s like to work on your team, something I believe is a key goal of the interview process. All it does is deliver a momentary synthetic dose of imperious adrenaline to Insolent Twerp (who shouldn’t be on your team anyway), to say nothing of an equally synthetic dose of homicidal impulse to the interviewee. Basically, this thing below, down to the blue hoodie:

It’s also why books like these make me want to puke. I’m well aware this goes against the grain of everything Google and other haughty giants believe in, but really: there are better ways.

Better ways

So, what then, discuss the weather?

Of course not. Technical screens are critical. It’s just there is a lot more to the job than writing code. Coding, as I have often said, is the “easy” part.

My approach to this as a tech leader has been, at a 10K foot level, relatively simple: Create as close to a real-life scenario as possible in the interview process.

There’s obviously a bunch of detail to unpack here (which I will, shortly), but it doesn’t change the fact that unless you’re writing kernel code or a search library (or of course CS 410), scratching up DFS code is unlikely to play any active role in your real programming life. And even if it does, it’s unlikely you will be doing it at gunpoint.

Logically the approach is simply a reframing of Will they be good at doing the job they are being interviewed to do? But it goes further: it gets the candidate thinking Will I enjoy working here? Fits go both ways.

Let’s look at a typical Day In The Life of a senior back-end engineer at, say, a fin-tech startup:

• 0830-0900: Coffee, watercooler, email catchup (two emails in there about pending PR reviews), Ooh there’s donuts in the kitchen!

• 0900-0910: Pod daily standup

• 0915-1130: Working with teammate on a design for the new ledgering service. Meeting with tech architects tomorrow

• 1130-1230: (Alleged) lunch break, interrupted half-way through her fish taco at 1152 by the prod alert comes in: Web-hooks are failing on payment service listener, HELP!

• 1152-1311: Pair-troubleshooting with DevOps, devolving fish taco in hand, combing through Sentry logs to see what’s going on. Some array being indexed out of bounds. Code doesn’t look familiar, something weird going on.

• 1312-1320: Tracking down the PR that introduced that bit of bad code

• 1325-1400: Scanning the newly introduced code, working on hotfix (because bad PR author is in a different time zone and likely asleep)

• 1400-1420: Pushing hotfix to prod, writing post-mortem on issue, publishing to internal wiki.

• 1420-1430: A little venting on Slack, finish eating now-deconstructed fish taco.

• 1430-1450: Walking around the block to clear head

• 1500-1630: Back to ledgering service design - got the deck ready, shared with pod and tech architects

• 1630-1715: Meeting with pod lead and couple of stakeholders on some new features

• 1715-1745: Catching up with PR reviews, sent a couple back with comments

• 1800-1830: More email catchup (some gripes from marketing that new landing page doesn’t render right on mobile, need to look into that tomorrow). Wind down.

Not an atypical day. You’ll notice that outside of the hotfix, which fixed someone else’s faulty code, there was hardly any net new coding during the day, let alone searching depth-first.

If this is the role you are looking to fill, what possible earthly signal did that binary search pseudocode on the whiteboard give you on your candidate, aside from some solid insights on her ability to write in a straight line on a whiteboard, and maybe her grace under pressure?

Make it a real technical screen.

Keepin’ it real

Let’s try a more realistic spin. Drawing inspiration from Typical Day above, here’s a stab at a “real” technical screen for the above senior (L3) backend engineer candidate, framed as a set of technical challenges.

Challenge 1: Can you help automate our reconciliation process?

We do a nightly reconciliation of our ledger with the ledger of our partner bank. The bank sends us a CSV file with all their transactions. Our transactions are stored in our MySQL database. Here is the <csv-format> and <table-structure>.

Currently this reconciliation is manually done by doing an export from our table and comparing it to the CSV received from the bank. We need to automate this to improve our operating efficiency.

Can you build a service with an endpoint that takes the CSV, runs the reconciliation and generates a report that looks like <report-format>.

You can use any language and/or framework you prefer, though Java or Python are preferred. Please state any assumptions you make clearly and provide a README to build and run your mini reconciliation engine. Bonus points if you Dockerize.

Challenge 2: Will you approve this PR?

This one is particularly my favorite - hence the name of the post 🙂

Peer code reviews are a critical component of our shipping pipeline. Here is a Github pull request that was submitted by one of your teammates. The PR has some comments that attempt to explain what it is trying to do.

Based on the diffs and comments provided, will you approve this PR? If not, why not?

Throw in some obvious and some subtle logic flaws, bugs and other issues in that sample PR. Style issues, null pointers, race conditions, infinite loops, that sort of thing. For more senior roles go deeper: throw in potential performance issues (e.g. double recursion without size-checking), security issues (e.g. unrestricted doc uploads without mime-type or size checks, unsalted password hashes, etc.)

(If it’s a front-end engineer role, I may throw in a Challenge 3 as well)

Challenge 3: Did the FE get it right?

Two images of a new landing page we are building for our marketing team are attached. Image A is a PNG export of the design from our marketing design team, Image B is our front-end engineer’s realization of the design.

Did the engineer get it right?

(Aka, testing the eye for pixel-perfectness).

Finally (and this is key):

You have 24 hours to work on these. Please email your solutions to <insert-email-address> by <insert-date-and-time-here>.

In other words, it’s a take-home challenge, with no Insolent Twerp eyes staring at you while you work. The posted solutions provide both the gating factor and the baseline material for the onsite technical screen.

To round this off, create a quantified rubric for the evaluation of solutions. Here is one I have crafted in the past for instance, for a senior back-end role:

Rank each on a scale of 1-5. If the overall score notches 3 or higher on a scale of 1-5, that could be a signal to bring in for an onsite screen.

Onsite tech screen

If the candidate clears the offsite challenge, the materials submitted provide a great baseline for the onsite screen. We use the time to discuss their solution, “collaboratively” make improvements to the coding solution, discuss alternative approaches, dig deeper into the code review, and many other things, for instance:

• Did she spot all the issues in the PR? How would she fix them?

• How can we make the recon engine more scalable? What if the CSV is 22GB? What if other banks have different formats, how do we abstract this further?

You would not only get a sense of the problem solving and raw code skills, but a good flavor of how she thinks, how she interacts with peers, how she takes feedback, or gives it. In other words - not just how good an engineer she is but how easy (or not) she is to work with. Is she “one of us”? Do you see her on our team?

And best part: vice versa. She gets a feel for work on your team too. If you really want to hire her, this is critical and invaluable. You owe that to your candidate, and it can actually be a great selling point too if you have a good engineering culture (more on culture in a future post).

Aside: In our increasingly remote-friendly and distributed world, “onsite” is becoming more metaphorical than literal. Platforms like HackerRank and CoderPad help bridge this gap quite effectively. We used the former quite productively at Yieldstreet.

But ChatGPT…

How would this pass muster in this era of AI-driven code generation?

Fair question, and to some extent unavoidable. But given that the posted solution is not the end of the story, rather simply the entry point to the technical screen, any “cheaters” would be relatively easy to sniff out in the onsite tech screen, where we dig deeper into their solution live. It is not easy to comment intelligently on “your” solution if “you” didn’t actually write it.

The aftertaste

We instituted a close flavor of this at Yieldstreet to great success - I was proud of the caliber of the talent we assembled there, the bar we set was exceedingly high. So much so that we actually received compliments, even from candidates that didn’t make the cut, saying how much they enjoyed the interview process. That’s a good feeling, to have left a pleasant aftertaste for the candidate.

Only one thing can sour it now - not getting back to the candidate in time. I was militant about communicating our decision to the candidate within no more than 2 business days either way. It is in exceedingly bad and unprofessional taste to ghost a candidate after an interview. It is actually unforgivable.

Now, write me some code for a doubly linked list. Go!

This is a post of exasperation. It may also be one of my shorter ones.

In a nutshell: that word, velocity, is killing me.

And somehow, within the sample space of product and tech leaders, I don’t think I’m alone in feeling this.

Thought experiment: here’s 3 fictional graphs:

All parameters held constant (same team, same toolchain, etc.), which one is your favorite?

I daresay many people would go with B. Up and to the right, right? Mostly?

For me it is C. And to be repetitive, within the sample space of product and tech leaders, I don’t think I’m alone in this.

Why? It’s the unit systems.

Unit economics

Before being swayed by the trends in these plots, it’s worth looking at the axis labels. It’s story points vs week (or some measure of time, like sprint).

This is the first source of my semantic frustration, and where the word velocity wreaks its unique brand of havoc. Per the Encyclopedia Brittanica, velocity is:

Units: meters per second, miles per hour, furlongs per fortnight: distance over time

So, armed with an unshakeable recall of high-school Newtonian mechanics, the powers-that-be seamlessly map “velocity” to “speed”. Improving velocity this becomes about shipping faster. Missed deadlines mean your team has a sub-optimal velocity. Le-sigh.

On the other hand, regarding “sprint velocity”, the Agile Alliance says:

Units: story points per week, story points per sprint: volume over time

In other words, putting aside the delightfully ambiguous voodoo of “story points”, it is a quantification of the team’s output capacity.

Aside: It’s worth noting here that the “volume” here is not necessarily code volume, but rather an abstraction representing functionality or complexity. A “5 point” story may take 5 hours of brainwork and yield less than 50 lines of brilliant code written in less than 25 minutes. That’s still a big “volume”. Yeah, it IS voodoo.

Either way, it has little to do with shipping speed. This is where Agile blew it on jargon. While these two have a (somewhat tenuous) correlation, they are not the same, at all. They answer different questions. Velocity answers “how much software can your team produce in time X”, as opposed to shipping speed, which answers “how fast can your team launch Y to production”. A bad question to begin with, but more on that later.

Tries to answer anyway: sprint velocity by itself is woefully inadequate to answer anything, being as it is but one of many many variables in the bizarro multivariate Markov Chain or Martingale or whatever Sorcery it is that attempts to make software estimation “predictable”. Channeling the old SAT/GRE prep days: Predictability:Stakeholder::Pipeweed:Hobbit

We love predictability, however illusory it may be. Works wonders for board decks.

The Illusion of Predictability

Swinging back to those velocity charts from before, this probably clarifies why I prefer C.

Unlike most other flatlines, this particular one is delicious. It implies consistency. Consistency breeds the fabled predictability. Flatlining tells a comforting story, that the team is finding its rhythm, working well together. That the people variable in the Markov chain is stabilizing, and we can look at other obstacular variables. Yeah, there’s no such word as obstacular, but there should be.

Aside: At this point I should apologize to all pure and applied mathematicians specializing in stochastic calculus for abusing Martingales and Markov Chains.

Let’s take this further, while we’re wandering in our Holy Grail fevered dreams. Here’s an even more idealized graph, with the team held constant:

Now my comforting story from earlier has become a heartwarming Christmas miracle tale that would make Frank Capra feel like Orson Welles. A healthy, predictable improvement in velocity over time. I’m positively salivating. I’m desperate to know what obstacles we removed that enabled the same team to produce more.

But here’s the catch: none of these.. velocitations.. will help me answer a common inbound question of the shape: “How long will it take us to ship X?”. Yeah, there is no such word as velocitation, but there should be.

My answer would still be the one that triggers white-knuckled fury in the listener aka stakeholder:

I can’t really say

Or if I wanted to temper it, sand it down, couch it, or apply some other metallurgical softening to it:

It depends.

“Depends on what?” they say. “Well, there’s a lot of other variables” I say.

At this point the battle is lost. Engineering is the bottleneck. Our velocity sucks. It’s the same story, over and over. Everywhere. No team is immune.

And it will continue being so. Why? Two main reasons:

• There are, as I said, other variables

• It’s the wrong question

Other Variables

Team capacity tells me how much, but not what. In a perfect world, Team A working on Problem X is wholly, solely, completely, consummately, uninterruptedly working on nothing else but Problem X, with no distractions and no external dependencies.

Except the last time that sort of focus happened was probably that term project for CS412 in your senior year (maybe), or the thesis you worked on in grad school, or, more recently, the MVP your 3-person Delta Force built in the garage to take you from 0 to launch and land your Series A.

In the messy real world of scaled commercial software development, your team will be yanked non-isotropically in several directions, often concurrently:

• Context switches. Production issues, emergency hotfixes, no-one-else-knows this-better-than-me-so-I-gotta-fixit (which is a different problem for a later post). The more senior you are, in tenure and experience, the more likely you’ll be yanked in this particular direction. This is unavoidable, and the most disruptive yank of them all.

• Debt, and unintended consequences. Touched briefly upon this in tech debt. The bigger the platform footprint, and older the codebase, the bigger this issue gets. Software is entropic. Something here will almost surely cause a weirdness over there. And you can’t estimate that upfront, it’s an unknowable.

• Internal dependencies. “Still waiting for greenlight on designs from Cassandra in Compliance”. “Ebenezar in email marketing is on parental leave, no one else knows what the dynamic variables for this email are.” “Pollyanna is still redlining the MSA”. Ring familiar?

  • Software creation is not the only activity that ships software products. Yet it’s the only one quantified and measured and held accountable. I’d like to see velocity charts for marketing or legal too someday. Or more ideally, a chart that tracks the work of the whole Problem X team.

• External dependencies. Yeah, you try and speed up the work of that guy in Nebraska (again). There are parts of the build we simply don’t have control over. ‘Nuff sed.

  • I really do like that meme, sums up “external dependencies” beautifully.

• Toolchain and process efficiency. Velocity measurements often end at Ticket.status = Closed. There is a good bit of journey left for code to travel before it hits production. And even more before it reaches everyone (if you’re doing things right with feature flags and blue-green deployments). Here’s a sprinkling of things to look at:

  • How efficient is your development environment?

  • How accurately does it reflect prod?

  • How fast can you run end to end tests? Regressions?

  • How long does it take to do a full build?

  • How quickly can DevOps provision a new cluster?

  • How efficient and automated is your shipping (CI/CD) pipeline?

  • How well are all the parties involved in the creation and promotion communicating with each other?

Bottomline: Velocity isn’t everything. Your team may be consistently outputting same volume, but it may be in service of 7 different things in addition to the one thing you’re trying to launch. Look at the whole pipeline for diagnosis and look beyond product and engineering.

Wrong Question

There’s the rub. No software team or leader has escaped being at the receiving end of “How long will it take to ship this?”

It’s the wrong question.

On so many fronts. It is inefficient, demoralizing, and ultimately unanswerable in any meaningful way.

1. It’s inefficient. Boxing the what and figuring out the how long is a losing game. All it breeds is exasperated exchanges that waste time and raise tempers.

   1. Plus you miss out on what software engineers do best. Hint: it isn’t coding.

   2. Dreading to tread into mythical man-month territory here (box, Pandora, etc.), but the refrain that often follows close on the heels of it is “can’t we put some more people on it?"“

2. It’s demoralizing. The question is transactional, not collaborative. Your tech team is not an agency bidding for the project. They are a part of a team that solves problems and moves the business forward. This framing makes them feel like code monkeys taking orders from stakeholders. No one likes working like that.

3. It’s unanswerable. Software, beyond a trivial point, is inherently inestimable. Any answer in the form of a timeline is either a massively buffered SWAG or a bare-faced lie (because Other Variables above). And yeah, the agency above is swagging and buffering as well, but they don’t have a choice.

   1. This is why I was never a fan of farming out bits of dev work to an agency, outside of well-bounded, well-boxed efforts that have a beginning and an end. Grudgingly at that.

So, is there a “right” question? Or is it all just wtf-land over here?

There is 😎. In fact, I mentioned it earlier in this post, in passing. Just turn things around a bit:

What can you build in N weeks?

To decorate it a bit more:

We want to build a product or feature to do X, or test theory Y, or improve KPI Z. Here are some initial thoughts on a feature. What do you think? What can we ship in N weeks so we can start testing and refine?

Here’s what this reframing does:

• You boxed time, not product. This makes actual mathematical use of that mythical velocity you’ve been measuring.

• You turned your software engineers into problem-solvers, because that is what they do best, not coding. Coding’s the easy part.

• You unified your product, design, and tech (and ideally other) teams into one problem-solving SWAT unit by bringing the tech team to the table. That is an empowering concurrent engineering move. Everyone is engaged, excited, collaborative, non-transactional.

• This was the whole damn point of measuring sprint velocity to begin with!

You have a much higher probability now of actually shipping something in the time box, or damn near close to it, because the engineers played a role in defining the what. While this may not immediately deliver results right off the bat, it will be the beginning of a much healthier and collaborative process. And it will define an engagement template that, if sustained, is virtually guaranteed to keep improving delivery timelines and quality.

Improving speed and velocity

Returning to the velocitous (another word that doesn’t exist but should) theme of this post, can we improve it? Can we get a given team, all things held constant, to produce more software over a given time? Aka can we get to that Christmas-miracle-tale velocity plot? And can we improve the speed with which we ship stuff?

Yes. Yes. And yes.

Let’s assume you have the right team assembled. Talented, team players, like each other, work well with each other, engaged, motivated, all that jazz. (If not, well there’s your problem.)

Once you have that team:

• Give them a problem to solve.

• Give them air cover.

• Get out of their way.

Over time, they will:

• Get better acquainted with the software and the toolchain

• Get better acquainted with each other, and find their rhythm

• Get more experienced individually (this is what career growth is for - it’s the most symbiotic phenomenon in a work environment).

• Get better acquainted with you, their leader(s). They will tell you what’s impeding them, what’s holding them back. They will tell you about the “other variables”.

Now just solve for the other variables 😉.

In all seriousness, this is actually where speed improvements come from once you have the right team and team dynamics. And while those are not easy problems to solve, they have proven solutions that just need a will and commitment (and often a budget) to execute on.

Specifically, in my experience, the single biggest disruptor to shipping speed is context switches. There’s enough there to merit a separate post, but the more you are able to enable uninterrupted focus, and channel your stabilizing velocity towards fewer things. the exponentially more you improve your odds of improving your shipping speed.

Which is not the same thing as velocity, dagnabit!

Post Factum

I guess the post wasn’t so short after all.

On a pseudorandom note, wanted to share a line in the book quoted above (who’s title I have borrowed for this post), circled it in #00d05c below:

Reminds me of a dialogue I have had way too often with stakeholders:

SH: When will it be done?
Me: We haven’t started yet.

Well, it is. But not just about that.

It’s more about the delicate waltz between software complexity and user experience - which often lapses into a violent tango, or whatever Patrick Swayze was doing in Dirty Dancing. And it is about something I wish I had done better in my many refactor initiatives over the years.

A few years ago, a pitch deck landed in my inbox. It was for a fixed-income investment platform: very specific, very niche, very confusing. Nothing was built yet; it was an aspirational deck (though perspirational is probably a better word for it). One of the slides in it had a proposed architecture diagram for the platform that looked roughly like this:

Ok fine, I coaxed that one out of Dall-E (hence the typos). But it is directionally aligned with what I saw. There were about 7 sub-systems, each containing north of 10 microservices, I think I counted about 19 data-stores, and a dizzying array of arrows that presumably represented data flows or message exchanges. And I’m sitting there wondering:

• Where do I begin?

• What if there is an error? Where in the name of Donald Knuth would you even start looking?

• What would happen if that went down? “That"“: any random arrow up there.

• Is there a wall large enough to contain the SPOG that tells you wtf is going on at any given time?

• Most importantly, how do you ensure a smooth user experience with a back-end architecture that would make Rube Goldberg blush?

I am not getting into a monolith vs microservices polemic here. For one, enough has been written on this topic. More to the point, I actually don’t have strong convictions either way, for the simple reason that I have seen both approaches work beautifully in some situations, and bomb with a theatrical flourish in others. The best I can say on which architecture I prefer is It depends. Or more succinctly: ¯\_(ツ)_/¯

But what I do have adamant views on, is user experience - and how that somehow starts taking a backseat as platforms get retooled for scale, a sacrificial lamb at the Altar of the Refactor Gods. And monolith → microservice refactors are most commonly deployed in service of scale.

SiaS

That’s not SaaS with a typo. That’s an actual acronym. It is now, anyway:

Software is a Service.

Software is a tool, a means to provide a service to some user or stakeholder. It is not the end. End users don’t see the wizardry of your architecture or the elegance of your code, and even if they could, they don’t care. If their experience sucks, your work sucks. And their experience is all that matters. Period.

Somewhere along the way of growth, we often tend to lose sight of that. This is particularly endemic in monolith refactors, inherently a very back-end heavy endeavor. An all-too-common narrative: startups grow, sometimes exponentially. Audiences grow, teams grow. To support all this hockey-stickage, the platform tries to scale, often awkwardly. Cracks start showing up in key flows. Inevitably, tech debt bat signals start flashing, fingers start pointing to “that monolith”, and a blueprint emerges to retool the platform for horizontal scalability by breaking it down.

This is all very good. I have been in these discussions many many times, often led them, and I have been consistently humbled by the engineering magic that happens in the room.

My main concern is: who else is in the room where it happens?

The room where it happens

Here’s a scenario.

The Buy Foo flow in your foo.com (or foo.ai probably, these days) has been keeling over a tad too often. That blocking call on the pricing API is bricking performance, the request backlog is piling up your Postgres connections, capping you at 500 concurrent foo purchases. Hardly web-scale.

So, a RefactorMoot is planned to make Buy Foo horizontally scalable. A legion of bright minds gathers in a room. A glance at the whiteboard in the room shows a dense graffiti of arrowheads and boxen and zigs representing module boundaries, and zags representing network boundaries, a bunch of cylinders for data stores. Something like this (or more):

You look around the room and see this collective, armed with coffee and Red Bull and sometimes stronger stuff, engaged in passionate debate, geekspeak flying around like verbal butterflies: “The foo purchase is massively I/O bound”, “Does it need to be strongly consistent?”, “Postgres is the bottleneck” (yes, it usually is), “Let’s split this out into a foo-payment service with a local projection”, “at least once delivery should be good” (shameless plug for this publication’s name).

Heady stuff. But then you look around the room again, and wonder: Where’s the front-end engineer who built the Buy Foo flow? The product designer who Figma-ed it? What about the product owner for the Storefront, was she looped in?

Most of all, is anyone here thinking of Janet over there, who’s waiting to buy some foo?

Almost all refactors for scale, microservices or not, involve the introduction of things like statelessness and async processing. Thread and network boundaries are introduced. Real-time is relaxed to “near-real-time”, a nice non-committal promise. Interactions are morphed from API calls to pub-sub event emissions. Eventual consistency is an almost-imperative for handling scale.

It is also, in my experience, one of the most difficult constructs to design good user experiences around. The back-end toolchain has gotten so rich and sophisticated, with concepts and frameworks like the actor systems, Kafka, auto-scaling, serverless, distributed tracing, and so on, that the realization of this refactor is not conceptually hard in that layer of the stack (execution is another thing). But retooling the UX in the wake of a scale-driven refactor without violently violating POLA remains stubbornly, infuriatingly challenging. (Aside: I love the acronyms in our industry. SPOG, SPOF, POLA… someone recently introduced me to git lola.)

See, when Janet clicks that Complete foo purchase button, she expects to see a shiny foo in her pending orders on the next screen. Strong consistency promises that. Eventual consistency promises not that: it literally says it will eventually show up. I have done multiple Google searches for things like “designing for eventual consistency” or “UX for eventually consistent flows” and have been, er, consistently disappointed with the results.

The point of all this is not to knock eventual consistency. I have nothing against it, it’s one of the many tools in the toolbox, a great one at that. My lament is the fact that Janet above is often an afterthought in this refactoring charrette. And by the time the ripple reaches her, it’s often too late. While the back-end hums along in its modularized emancipation, and Grafana hums along in the comforting greenspace of passing health checks, Janet proceeds to get a mild coronary on clicking that Complete Foo Purchase button, because she’s just been deposited on her My Orders page and where the hell is the foo she just bought? Or she is rightfully peeved with a confirmation page that displays something like Order ID 9a3dfcf9-1d1b-423c-8f43-96e25fee73a7 submitted, above the fold, and a Check status button she has to keep clicking. All because that’s all you had time to shoehorn in after the scale refactor made that flow eventually consistent, and you never did loop in the front-end engineer did you!

In other words, few people seem to be giving an indented hoot about changes to Janet’s new post-refactor foo-buying experience.

I really need to stop saying foo soon, but bear with me a bit.

Concurrent engineering

So, what do we do?

I was planning to write about concurrent engineering in a later post, perhaps the only meaningful crossover from my grad school research into my professional career. But worth introducing here. It’s an engineering practice that emphasizes concurrent consideration of multiple aspects in the design of a product by giving voice to multiple perspectives at once.

Put simply: Get all the right people in the room!

Get the front-end devs, get the designers, get product, get QA. Treat the refactor like the new product build that it is, up and down the stack. Most importantly, start the discussion with Janet. How does this affect Janet’s foo-buying experience? What will change? What can we do to make it not suck for Janet?

Because remember, you’re doing this refactor for Janet. Not for anyone else.

The scene unfolding in the room looks different now. Channeling my movie geekdom, let’s try this as a 2-minute short. Any WGA format non-compliance is unintentional, and casting is left as a thought exercise for the reader.

There’s a movie I’d like to see. Wish I had done more of that.

Mercifully, this is the point where I stop saying foo.

Elegance, not perfection

At Yieldstreet, we had our own set of core values that I drafted for the Engineering org, additive to the company-level ones. One of those was this:

This summarizes my hard-earned lessons on too many refactor efforts over my time, especially that last bullet. There was a lot of heartburn over the years, across multiple startups. It had to be codified.

Aside: our values all had this “X, not Y” syntax, because why not. For those curious, this was the full set:

Post scriptum

My search for elegant UX paradigms for eventually consistent systems goes on. Would love to know what you have done, or come across, that addresses this gracefully.

It is also quite true that not all refactors, for scale or otherwise, have a direct impact on end human experience. Not all software even has a user interface - API plays for one (though I would argue that the idea holds true in some form there as well). Conversely, there may other refactors, not involving eventual consistency, not related to scale, that may still have a UX impact.

The point is to start refactoring discussions with the consideration of end user experience impacts, even if to establish right up front that there aren’t any. And that requires getting all the right people in the room where it happens.

It’s a pretty common cost-saving measure in those early cash-strapped days. Nothing wrong with that — the planet is studded with talent-rich geolocations like Latin America, Eastern Europe, South Asia, where you can get a seasoned Python geek for an annualized cost of KKR’s weekly budget for Post-Its.

But it seems most of these startups, while they notch material $-savings with these “offshore teams”, can’t seem to stop unspooling into jeremiads of why it isn’t working for them. “Communication is a nightmare”, “Shipping is delayed”, “Keep losing team members”, “They just don’t get it”, “Gotta deal with all this tech debt now”. Not all of these laments are misplaced either. I have seen it not working more often than working, my own experiences included.

So: is there a point here, or am I just adding my voice to the irritated chorus?

A bit, yes. But with a counterpoint: when it works, it is magical.

More on that soon but let me start with an instance of one of my groaners on this.

San Francisco, 2004. I was working for a fintech startup called Finaplex (a name who’s I’m Feeling Lucky goog-hit invariably pulled up a bovine steroid, along with a "Do you mean..?” rider), in what would roughly be the Mesozoic era of fintech. Finaplex was really a trailblazer. The Finaplex Wealth product was a sprawling white-label wealth management platform for bulge-bracket financial institutions. Our clients included Lehman Brothers, Bear Stearns, First Republic, to name a few (all coincidences are coincidental). My client at the time was an online brokerage called HarrisDirect, which was subsequently acquired by an online brokerage called E*Trade, which was subsequently acquired by an on/off/every-other-line brokerage called Morgan Stanley.

But that’s beside the point.

The Harris project was huh-yooge. It had about 17 sub-projects, for account aggregation, IVR, proposal generation, market-data integration, portfolio accounting, and many more. At one point, Finaplex decided to work with an “offshore” firm in India, whose name shall remain hidden (mainly because I’ve forgotten it, so let’s call it Y, to not confuse it with X).

That. Was. A. Nightmare.

For one, Y had a tendency to deploy a mid-sized republic of engineers on efforts that 2-3 senior engineers could have easily brought home. Compounding that there was a large set of revolving doors, so we could never get any notion of continuity across the 37-odd engineers let loose on, say, a reporting tool.

Secondly, there was literally zero overlap in working hours between San Francisco and Hyderabad. Word docs or titanic PDFs were hurtled back and forth across 13 time zones, multiple times a day. This was pre-Slack/Teams/Zoom/all-other-RT-collaboration tools.

But most importantly, there was the Mothership Syndrome. Overlords in San Francisco delegating to legions of offshore minions. Mostly (but not always) intentionally, mostly (but not always) subconsciously.

In aggregate, this was a nightmare, a bucket of worry for all - so much so that our friend and colleague Jochem, front-end dev to the stars, morphed the Finaplex logo into a new one called Chintaplex (chinta being the Hindi word for worry).

So that sucked.

But the counterpoint…

One of my most thrilling experiences as a tech leader has been building and managing what by conventional definition would be considered an “offshore” engineering team, spread across 4 countries and 3 continents: our team at Yieldstreet: New York, Brazil, Argentina, Malta (and eventually Utah, Oklahoma, and Boston as well). And, for the most part, this collective worked together like a multi-lingual cross-cultural symphony that rarely if ever devolved into cacophony and built some of the most amazing software I’ve seen.

So why did this work, whereas Y above (and a few after that) didn’t (quite)? In theory it was simple: we engaged with them exactly as we did with our NYC-based engineers.

I believe the reason many if not most traditional outsourcing models falter is rarely that the offshore engineers or their companies suck. Y had good engineers, as did most of the non-US based teams I worked with over my time. In fact, more than good, brilliant engineers, from whom I learned a lot.

The fault, more often than not, lies in the hiring company. Or rather, the space between, aka the engagement protocol. An implicit pecking order grows unchecked, an us-and-them divide. Little or no agency is given to non-local teams. Just do it as you’re told, if not the actual words deployed, is often the covert (sometimes overt) sentiment. As the corporate platitude goes, they are not set up for success.

This in fact ends up doing a disservice to the business as well. We miss out on all that this pool of talent has to offer by reducing it to a code-monkey army. I mean, your technical architect in Buenos Aires may draw the same comp as your analyst in New York, but that doesn’t make her an actual analyst. So why would you limit her responsibility scope to that role? Don’t you want to let her brilliant architect brain loose on your tech challenges?

This is what worked for us at Yieldstreet: avoiding that.

There are many details in this ongoing experiment, but it really boils down to 5 things, that all coalesce into one idea: They are us. We’re one team.

1. We stopped calling it offshore

That was Step 0. We just banished that word (and adjacent ones, like outsourced) from our vocabulary. Decades of sub-optimal engagement style has actually morphed these terms into borderline pejoratives due to the implicit pecking order they’ve come to suggest, if not enforce.

These were our global teams. Any us-vs-them reference, however oblique, from anyone, was not tolerated. This seems like superficial verbal fluff, but it’s not. It was a very important distinction, and required unlearning of old habits, on both sides.

And it wasn’t just words. We meant it.

2. We integrated the teams

In every way possible. An engineer is an engineer is an engineer. Regardless of where she is located.

• All pods have a geographical mix of team members, tailored for the pod’s charter and needs. Aka the mix is merit-based.

• Roles are similarly distributed on merit: pod leads, technical architects, manager, all others. Again: merit-based. We even had cross-office reporting structures, though there is some benefit in keeping those as local as possible.

• Everyone has agency. Everyone plays a role in shaping what is built and is not just building what they are told. Everyone is encouraged, even required to ask questions, including the most fundamental question often denied to traditional offshore teams: Why?

• Team members, regardless of office location, interact with other parts of the org as needed - especially internal users of the products they work on: marketing, customer support, operations.

One thing to watch out for here is time zones. The whole “24-hour dev cycle by getting a team in Asia” thing is an effing mirage. You need a meaningful overlap window in your workday for this to work well.

3. We folded them into our HR processes

This was key. All team members, agnostic to their location, were on the same career development plan, had the same L&D opportunities, same review process, same bonus schedules, same recruiting process, same responsibilities based on their role, same social ceremonies - holiday parties, summer outings, dunk tanks, that sort of stuff. We even provided them with healthcare benefits.

Each location had Yieldstreet offices where these teams worked, following the same hybrid schedule, and complete with company branding, protein bars, and, well, other kinds of bars.

4. We gave them skin in the game

A specific innovation that built on #3: They have equity in the company. They are on the same ISO plan like all other employees. This is the first time I have seen this done, props to Milind, one of Yieldstreet’s co-founders, for instituting that. Same grant brackets, same vesting schedules, same incentive, as everyone else.

5. We enabled frequent facetimes

We put in a cross-office travel budget; a mere fraction of the “savings” achieved by the geo-distribution of our talent. Multiple times a year, cohorts travel across our offices for a week or two, work side by side, eat and drink side by side, and actually have an enriching bonding experience and cultural cross-pollination.

Trust me, if you haven’t heard Maltese spoken with a Portuguese accent at the tail end of 4 pints of Cisk, you haven’t lived.

What’s more, these are now, I hope. friends for life, on both sides of the Atlantic. And they carry a strong identity of being a “Yieldstreeter”.

Caveats

I guess in hindsight I could have just reduced this whole post to one line: Treat your non-local teams the same way you treat your local teams.

That said, I do realize there may be legitimate gating factors to this: company policy may not allow full integration, regulatory guardrails or internal governance policies may limit data sharing etc. You will need to tailor the specifics to your context. But that doesn’t change the underlying spirit of the idea, which is human and behavioral: one team.

It also requires the right kind of offshore partner, one that is open and flexible to adapt to your desired working model, whose business strategy is not volumetric, where 5 engineers are deployed to move a button 5 pixels to the left. I’ve been lucky in this regard to find good partners. (Y was not one of them.)

Of course, all this presumes you treat your local “FTE” teams well. I hope you do.

PS This journey was not always easy. There were definitely speed bumps. Cultural nuances and language barriers sometimes triggered misunderstandings. Pecking orders did evolve despite best attempts to avoid them. And it is still an experiment in progress. Like most everything else. But it has worked better than any distributed model I have seen or managed. And it keeps getting better every day.

Back in June 2000, I was writing some software for the Private Placement Division of a merchant bank called Thomas Weisel Partners in San Francisco. JDK 1.3.1 if I remember right. I was quite proud of the stuff I built there, especially some third-party integrations with vendors like FedEx and Siebel (remember them?) - all this before things (or even terms) like “API” existed, let alone REST. 

But one piece of code I was particularly proud of was a multipart file uploader, which I pretty much wrote from scratch over an all-nighter by reading and implementing the original RFC for it. Because there was nothing already available to do it - or rather, there was no Stack Overflow to check if there was. It was probably not very elegant, but the damn thing worked - on the first try. And Private Placement moved a lot of docs around, so it was “mission-critical”, and needed to be maintained. 

A few months later I discovered Struts and realized I could have slept in the night I wrote it. ¯\_(ツ)_/¯. We never got around to moving that functionality over. It’s probably still being maintained by some poor schmuck in Nebraska. (I’m still proud of that bit of coding though 🙂).

Point of all this dorkage? All tech is debt. It’s only a matter of time. 

“Tech Debt” is the eternal bugaboo of startup-land. Pegged as the ultimate velocity killer (more on “velocity” in a later post), the persistent itch in the build-and-ship pipeline that just can’t get enough scratch, the receiving end of most accusatory fingers when bugs happen, the one thing standing in the way of continuous delivery utopia. Techies underscore it to clamor for sprint-space to do their beloved refactors (I’ve done it), rationalize missed deadlines (more on “deadlines” in a later post), and warn of dire unintended consequences and collateral damage if it isn’t addressed now. Business is confounded by it, and rightly so.

The way I have written the above paragraph makes it seem like it’s all resplendent baloney. It’s not. Tech debt is a real thing, and left unattended, it can cause all of that above. I have personally experienced this pain, at least twice. I just believe there is more nuance to it than the blanket statements I have seen hurled at it. 

Here’s three things I have come to believe:

1. Tech debt is inevitable. 

2. There’s good tech debt and bad tech debt

3. Good debt can decay into bad ← this is where the shyte goes down

Tech debt is inevitable

Probably a good place to provide at least my definition of tech debt. In the broadest sense of the term: 

Any piece of software that has to be maintained over time is, technically speaking, tech debt. 

AKA:

All tech is debt

Software ages fast. And unlike many cheeses, mostly not well. 

To be sure, there’s the voluntarily incurred debt, stemming from things like shortcuts on one end to over-engineering on the other, or “organic debt” in the extreme (Appendix I). But even if you avoid all of these traps with your spectacular foresight and craft that perfect async service, that distributed framework you built your service on top of may be EOL-ed (or worse, licensed – true story), rendering your gumdrop software elegance into a mass of debt with a ticking clock inside it. Or the use case you architected for is no longer relevant, and you’re stuck with 4 microservices, 3 Kafka topics, and 11 events to update a user profile, which felt right at the time, but a CRUD endpoint would now do just fine. 

Ergo: tech debt is inevitable. That’s just the way it goes.

(Except maybe for that use-once-and-toss data migration script you whipped up in Rust, that no one will ever see or inherit.)

There’s good debt and bad debt

Tech debt gets a bad rap, because “debt”. 

But if you’re not voluntarily incurring a little tech debt in your startup journey, you’re most likely not moving fast enough. Or, as the bromide goes, “perfect is the enemy of good”. If a little bit of hardcoding (with a plan to abstract it out in the next couple of sprints) gets you to market sooner, or beat the competition to that launch, that’s the win. You don’t want to end up writing the best piece of software that no one uses. 

There are of course, empirically bad slabs of debt often incurred by startups in early stages. I mean, don’t build your entire platform in Clojure, for Pete’s sake – where will you find the developers?

However, a small manageable piece of debt, voluntarily incurred to move the business forward faster, is a good thing. However, it is only as good as the payment plan built in to work that debt down. If you’re smelling a credit card debt analogy here, it’s coming soon, as this segues well into the next point.

Good debt can decay into bad

Almost all bad cases of tech debt I have seen, encountered, dealt with, even created, have been, in some shape, a case of “good debt” devolved into “bad debt”. This is the kind that causes a newly hired CTO to beg for “3 months to clean it all up”. And what early-stage startup has that kind of time?

Cue the credit card debt analogy. We all build debt monthly on our cards. It helps build our FICO scores and credit history. And the financially well-behaved among us pay the statements down in full every month. Conversely, if left unattended it can rapidly become untenable. 

And so it goes with tech debt - the paydown plan is not put into place as the debt is being created. Or it is created but not followed. Or attempts to follow it are shot down. Or all of the above. So, more debt is created on top of old debt. And the debt-stack Jenga continues until the only solution is filing for bankruptcy. Aka the dreaded “full rewrite” - which causes engineers to salivate and business to hyperventilate in equal measure, both for good reasons. 

If there’s one thing I’ve learnt, the hard way, it’s this: Good debt without a paydown plan will inevitably go bad. Keep working the debt down every sprint, bit by bit, even as you accumulate new bits of debt.

Where do we go from here

I promised not to be very prescriptive in these ramblings. I won’t. Also, because historically, I have had limited success in managing this particular challenge. So, what follows here is really commentary on what I tried to do to contain it, or what I wish I had done sooner.

First off, avoid organically bad debt. A good tech leader/CTO will do this by default: stack selection, foundational architecture, etc. I won’t spend too much time on that here for now. 

Most importantly: Build a payment plan along with the debt. This can take several shapes but is mostly about habit creation and expectation management. 

1. For every ticket you create in a sprint that is “tech-debt-y” (say a declarative hardcoding for something) create a companion ticket for Sprint N+1 (or 2) to refactor to something less crude. 

2. Fold debt reduction into ongoing enhancements. It will always be “faster” in the short term to ship something by stacking debt. Instead, price in a couple of extra hours to work that old debt down. Best way to shave tech debt is in the process of building something new.   

3. Planned obsolescence. Remember I said all code eventually becomes debt? A large maintenance surface area can cause serious drag on shipping speed and increase the chance of unintended consequences. Keep measuring what’s being used and what isn’t, sunset old bits of code on a regular basis. A key prerequisite here is to measure use and adoption of everything new you build of course - but good product orgs do this by default. 

4. Despite all best efforts, it is quite likely some corner of your platform may morph into a Golgothan over time that needs to be neutralized. It is more likely than not to happen. Define a quantified framework (complete with OKRs) to rationalize taking on this larger tech debt work. E.g. in the past, I have formulated a “PBSD Framework” for greenlighting and scoping large-format debt reduction work. It helped - at least to make the case for it.  

I could go on and on about this, but I should stop here. It’s already gotten too long. If there’s interest, maybe there’s a part deux here someday. 

But TL;DR here: don’t knock tech debt. It’s not all bad, and it’s not your fault. Just keep the Golgothan away.

Appendix I: Organically bad debt

I don’t know what else to call it, but this refers to debt that’s bad from the get-go, unlike “good debt” gone bad over time.

• Prototype Becomes Product. Very common, sadly. And has happened to me, twice. The little proto you built to demo to VCs for your seed round? That’s not your final product. Use the proceeds from the seed to build it for real.

  • This is an extreme case of “good debt gone bad”, and in absence of good tech leadership from the get-go, will almost certainly require a full rewrite. 

• Obscure Stack Debt. Mentioned before: don’t go Erlang or Clojure or (shudder) Scala.  Stick to well-understood, well-supported, talent-available languages and stacks for your core platform. Or you will have to rewrite it anyway. There’s always peripheral stuff you build to try out new cool tech. 

• Complexity Debt, aka over-engineering. Simple is beautiful, Microservices aren’t all that. Start with the simplest flow, complexify from there if needed (and in most cases isn’t). Making a complex distributed architecture simple is often harder than incrementally layering complexity with the right foundational architectural decisions. 

• I’m sure there are many more. These are the ones I ran into the most. Would love to hear about yours!

Appendix II: The PBSD Framework

I’ve used this in the past to both analyze/greenlight and then measure the impact of large-format tech-debt initiatives, like major refactors or full rewrites. This was the PBSD rubric:

• Performance - is it slowing stuff down?

• Bug frequency - is it causing too many bugs?

• Scalability - is it causing bottlenecks under traffic?

• Dev velocity - is it slowing down dev/ship velocity?

For each proposed tech debt work, make it mandatory to publish a PBSD memo for discussion:

Function

<describe the function supported by this bit of software>

Cost of debt: 

<some PBSD score, e.g. “PS”, to indicate the debt was impacting performance and scalability, along with some description of how>

Complexity: <numerical score from 1 to 5, increasing with level of complexity>

Debt reduction path

<a bulleted list of proposed path to work down the debt>

Success metrics

<how would we measure success – OKRs, in effect>

It’s catchy. Always a good thing. But there’s something to parse in there beyond delivery semantics. It’s a, well, catchy lede for what I am thinking of rambling about here - tech challenges in early and scale-stage startups. Because whatever challenge (or few) you may be running into, it’s very likely I have run into it… at least once.

See, I’ve spent the last 20+ years in #techstartupland - mostly in fintech (but that’s parenthetical) - but entirely as a geek: a software engineer, an architect, an engineering manager, a tech advisor, angel investor, and particularly, over the last 15+ years, as an early-to-scale-up stage CTO.

That’s a chunk of time to stumble, scratch, burn, recover, get bricked again, recover again. And learn, at every tumble, stumble, topple, slip, pick-your-fall-verb.

The learning never stops.

I may not have too many answers (also pontification is not my thing, nor is prescription), but I sure will enjoy sharing my experience on specific problems which, across my time in tech, I’ve realized are ubiquitous. domain-agnostic, and in most cases, portable… as are some of the remedies that worked for me.

So that’s the idea. To bloviate about (stop me if any of these give you deja-vu) tech debt, engineering culture, offshoring, the infamous “productivity” mystery, strange new-age-y (for me) buzzwords like “full-stack”, stuff like that.

It’s likely you’ve run into at least one of these, at least once.