Anthony Accomazzo's musings on technology and the mind

SpaceX's 5-step Engineering Process


SpaceX launched from Santa Barbara County the other night. Their flight path was the closest it's been to the West Coast. I was out out on the beach to enjoy it.

The most amazing part was that the second stage was high enough that it was awash in sunlight. The contrail of the rocket was lit up, creating an unbelievable streak of light in the night sky:

<div style={{width: "100%", textAlign: "center"}}>
<img
src="/spacex-engineering-proces/second-stage.jpg"
alt="Second stage of the SpaceX launch, contrail lit up against the night sky"
style={{margin: "auto", maxWidth: "90%", maxHeight: "350px"  }}
/>
</div>

So, I got into a SpaceX kick. I watched some videos and stumbled upon something special. In a tour of Starbase in Boca Chica, Elon Musk [details SpaceX's 5-step engineering process](https://www.youtube.com/watch?t=794&v=t705r8ICkRw&feature=youtu.be):

1. Make your requirements less dumb.
2. Try very hard to delete the part or process.
3. Simplify or optimize.
4. Accelerate cycle time.
5. Automate.

I can tell these steps were forged through years of pain. There is so much wisdom here – not in the steps themselves, _but in how the steps are ordered_. They are designed to fight against the natural bad habits and instincts of engineers.

Musk says something that should be relatable to all of us: "I have personally made the mistake of going backwards on all 5 steps, multiple times. I automated, accelerated, simplified, and then deleted."

He then goes on to describe a very memorable example of a time where he ran these steps backwards. (The example is from Tesla, where I presume he echos the same steps.)

### Fiberglass mats

In the battery pack production line, there was a stage to install fiberglass mats. These mats were installed on top of the Model 3 battery pack. This stage was choking the line, which in turn was choking the entire Model 3 production program.

As the stage was already automated, they decided to invest more into fixing the automation. Can we automate more of the installation by making the robot better? What if the robot moved faster? Can we give the robot a shorter travel path?

That helped, but didn't fix the problem. So next, they optimized. For example, they were spackling the entire battery pack with glue in order to attach the mat. What if instead of spackling glue on the whole pack, they just put little dabs on it? The mat was eventually sandwiched between the floor pan and the battery pack anyway, dabs should be fine?

But these optimizations weren't enough to clear the bottleneck either.

So finally, after automating then simplifying then optimizing, Musk asked: "What the hell are these mats for?" Blank stares.

He went to the battery safety team first: are they for fire protection? "No, they're for noise and vibration."

So, he went to the Noise, Vibration, and Harshness (NVH) team. Are these mats for noise? NVH: "No, they're for fire safety."

Nobody knew why the mats were there and who made the requirement. They decided to run cabin noise tests with and without the fiberglass mats. There was no difference.

The mats – which had costs millions to automate and millions more in production line delays – were useless.

### #3: Simplify or optimize

The most important part of the 5 steps is that "simplify or optimize" is the third step. No engineer needs to be told to optimize. It's in our nature. It's what we're trained to do. But, the most common error of a smart engineer, Musk argues, is to "optimize a thing that should not exist."

Why do engineers fall into this trap?

Musk attributes this to our training in school. Everyone has been trained that you have to answer the question. You can't tell your professor: "I don't want to work on this problem, I don't think it's relevant."

I think there are even more dimensions to it.

First, there's so much dopamine one gets from optimizing a thing. It's one of the most satisfying parts of the job.

Second, requirements are often hidden. Requirements don't just live in specification documents, architecture diagrams, or tickets. Requirements are often more pervasive and less obvious. They're already _integrated_ into the design.

In Musk's battery production line example, the fiberglass mats _were just there_. They're a bad requirement now, perhaps they were a good requirement at the beginning. But the mere presence of the fiberglass mats on the line _is an implicit requirement_. There's inertia, which is keeping the fiberglass mats in the car and on the line.

It's difficult to constantly question things, easier to keep the momentum going. So, when the mats run into problems, engineers rush to fix the problem, because that's what we engineers do. When engineers redesign other parts around the battery pack, those designs implicitly contain the requirement of the fiberglass mats. Because, well, they're _there_.

### #2: Try very hard to delete the part or process or wool sweater

The "bias in engineering tends to be very strongly towards 'let's add this part or process in case we need it.' But you can basically make 'in case' arguments for so many things."

That part I don't think is an engineering thing – that's a human thing!

I'm reminded of Marie Kondo, casting her judgment upon me as I pull that gray wool sweater I've never worn out of its unvisited corner in my closet. It itches (am I allergic?), it's never fit quite right, and there are basically 3 days a year where it's cold enough in LA to wear it. _But maybe_...

"In case" has buried hoarders alive in their own homes. Kondo helps people throw things away by giving them little rituals to make it feel OK. (I thank my gray wool sweater for that one hot, itchy night at a New Year's party a few years ago then give it away.)

Musk likewise has a rule to help his teams delete the cruft. It appeals to the analytical nature of his engineers:

"If you're not adding stuff back in 10% of the time, you're not deleting enough."


Unearned wisdom (don't turn off the lights)


Our team did an escape room together recently.

Before entering the room, the host prepped us. You could tell he was trying to help us avoid common cul-de-sacs. He gave us pointers that would ensure we wouldn't get stuck in situations that would often consume a team's time.

His last piece of advice was also the most cryptic: "You're going to get to a part of the game where you're going to be tempted to turn off the lights. Don't do it. It's a waste of time."

We nodded, perplexed – easy enough.

He continued: "In fact, even though I've told you not to turn off the lights, you're still going to do it. Everyone still does it. Again, you shouldn't - it's a waste of time."

_Pffft_! So silly that no one follows this simple advice. We won't be the foolish team that fumbles around in the dark. Advice received.

About 30 minutes into the escape room, we found ourselves at a juncture. Without getting into too much detail, we were stuck, and had just unlocked an arsenal of black lights. We began painting the room with the lights, looking for clues. All of a sudden, it felt _very relevant_ to turn the lights off.

The team debated:

"...should we turn off the lights?"

"No, he specifically said we shouldn't do that."

"Are we sure he was referring to this room and situation? He could have been referring to the room we just came from."

"He said not to turn off the lights."

"Maybe there's another upcoming situation where that advice applies? We just got _black lights_!"

"It couldn't hurt to turn them off for a little and just _see_. We're stuck."

So, we turned the lights off, and fumbled around in the dark for 5 minutes. Twice.

And, of course, as the host warned, turning off the lights got us nowhere.

Carl Jung advised to be weary of unearned wisdom. You can receive all the advice in the world, but so often you have to reinforce that advice with your own experiences.

Advice can't help you avoid all the pitfalls – you have to go through it and see for yourself. Only then, with experience and first-hand anecdotes, does the advice transition into wisdom. Only then can you spot situations and appreciate that the wisdom applies (vs fooling yourself into thinking "this situation is special").

While advice can't always help us avoid hazards, at least it can shorten our time fumbling around in the dark.


The metaverse is for remote teams. Is Meta ready?


Out of the cornucopia of 🌈 possibilities 🌈 in Facebook's metaverse "vision video," one use case stood above them all: Enterprise.

That's right: the first thing this author wants to do when jacked into virtual landscapes with no limitations is meet with his team for a daily standup. So I ordered an Oculus and downloaded _Horizon Workrooms_. After years of failing to enter the workplace, Meta's latest effort is to _be_ the workplace.

This is a colossal push: they've not only re-entered hardware, but have created a state-of-the-art VR headset. They've designed a virtual world, complete with hand-tracking, lip-syncing, and a whiteboard.

_Horizon Workrooms_ is early and while it's yet to fully realize the potential of immersive VR meetings, there's enough here for small teams to enjoy.

But as novel as this entire stack is – down to the name-change to Meta – this experience has a giant catch lying in wait for whoever attempts it.

More on that later. First, I want to advocate for why I believe that the promise of VR is that it provides a Goldilocks point for coworker interaction for remote teams. Somewhere between Slack and a real-life boardroom – and _way better_ than a Zoom:

**You're no longer on camera**

In Slack or email, almost everything about an interaction is stripped away. You send and receive text, which are ideas in their most raw form.

Today, we consider a video chat a "halfway point" between Slack and a real-life meeting. After all, you can see the person! You can see their facial expressions and hand gestures, hear their tone.

But a videochat is a strange conversational derivative. You're not so much talking to each other as much as watching each other talk to each other. You're watching someone on camera and you're being watched on camera. Eye contact is difficult if not impossible given relative camera/display placement.

Because you're on camera, you have to be still. You can't change your body position much. And you have to keep your head and eyes locked towards the screen/camera.

Further, the meeting location is rather intrusive: so many of us were reorganizing our rooms at the start of covid to hide the fact we were inviting cameras into our bedrooms.

**Better conversational flow**

Perhaps the worst part of video chat is that it's hard to find a conversational flow in a video meeting. I believe the primary reason is that it's hard to deliver and pick up on social cues that would allow for cordial interjections.

With eye, mouth, and gesture tracking, in VR we're able to reclaim all the social cues that allow conversations to flow. If I say something that makes Wendy light up with a thought, I can see it, cast my gaze to her, let her interject, then proceed seamlessly.

(Latency will still be a factor, but low-latency video games like first-person shooters offer a lot of hope about the potential.)

**Meetings that energize, not drain**

There's a sense of _presence_ in VR that's hard to describe but immediately palpable. My sister came down with covid for Christmas. In lieu of seeing her, we hopped into VR to play mini-golf together ([cannot recommend enough](https://www.oculus.com/experiences/quest/2462678267173943/)). Oculus' tracking is currently rudimentary: you get head tracking and basic mouth tracking. But even with those basics, it was so striking: it really felt like _her_ animating this virtual character!

This is what VR enthusiasts call _presence_. This sense of presence is due to a variety of factors:

First, and most obviously, the world is taking up your entire field of view. You are immersed.

Second, thanks to tracking sensors, your friends and colleagues are _embodying_ their characters. So it feels like they're in the same room with you (which is never a feeling you get on a video call).

Third, like a regular meeting, I'm accountable to where I'm placing my attention. What's more, I can _feel_ when attention is on me (and when attention is wandering).

Last, because you are _in_ the space, you can move about the space naturally. You can pace the room. Instead of giving a presentation glued to my seat, I can walk around, making the presentation more dynamic for both me and my audience. My head and my gaze can move about and sweep the room of people. (VR headsets today have smart features that prevent you from bumping into your furniture.)

We're still missing great facial expression tracking and eye tracking in VR. But already the tech is close enough: in my experience so far, virtual meetings are (finally) energizing as opposed to draining.

**You get to leave some personal stuff out**

Just as important as what one _gets_ in a virtual meeting space is what one gets to _leave behind_: your appearance, bad hair day, messy wardrobe, makeup, cluttered living room, kids bursting into your video call background.

Leaving quarantine for the office, it's striking how much about office life can be _too_ personal: it is hard to hide everything from your dietary habits to your bowel movements from your coworkers.

For many kinds of social interactions, VR won't cut it. What's a first date without a first kiss?

But for enterprise, it has the potential for the perfect balance of intimacy and distance between team members.

**A place to do work**

In the short-term, I see companies using VR to come together for acute events, like meetings or hangouts.

But in the long-term, I see it being a space that employees occupy more and more often, even if they're not meeting with anyone in particular.

With Meta's Horizon Workrooms – Meta's first stab at the virtual office – you can already bring your keyboard and monitor into VR:

<div style={{width: "100%", textAlign: "center"}}>
<img
src="/metaverse-1/workrooms-cap.jpeg"
alt="bringing your keyboard and monitor into Horizon Workrooms"
style={{margin: "auto"}}
width="100%"
maxWidth="500px"
/>
</div>

But, the resolution of your virtual monitor isn't very good, due to limitations in the resolution of the Oculus headset.

I've perused some VR YouTubers that are exploring the frontiers of the technology here. There are some [headsets that blow their minds](https://www.youtube.com/watch?v=NC-puJQRVGI). These headsets are currently prohibitively expensive, but of course it's only a matter of time. (Lots of great reaction videos out there – [love this guy in a flight sim](https://youtu.be/e6djDSf0kxg?t=885).)

Once your headset resolution is good enough, there's little stopping you from having three 4k displays in your virtual office. And the setting for your virtual office can be atop a mountain in Steamboat Springs, Colorado or overlooking La Rambla in Barcelona.

This is why I can easily imagine a world where I spend several hours per day "heads-down" in my virtual office.

On a personal level, that feeling of "transporting" to the place where you do work is very helpful and focusing. When I moved from working out of my bedroom to working out of a dedicated home office, I saw immediate benefits to focus and productivity. VR promises to give every at-home worker that same benefit, perhaps even to a greater extent. My company can render and furnish a virtual office for me to work in that _feels like_ the team's HQ: company logo/paraphernalia, photos, inside jokes, metrics dashboards, etc. This sense of place reinforces the idea that I am a member of a team, part of a greater whole.

On a team level, I anticipate greater camaraderie as customized spaces become familiar haunts where the team brainstorms or passes the time together.

**A place to socialize**

Speaking of passing the time: who's ever used that phrase when referring to a video call? Polling my friends, the "Zoom hang" is a dreaded approximation of the office happy hour. It feels more like a chore than a fun, engaging social activity.

I still need to experiment with how after-work socializing in VR scales. But having social games with almost zero learning curve seems like a great way for the team to hang out while actually having fun. You can play a round of mini-golf or a round of your favorite board game, and thanks to improved social cues everyone can feel included.

That's all the potential. So, today:

### Where does Meta fit in to all of this?

Meta is trying to advance both consumer and business fronts at the same time. As I've made clear so far, I'm bullish on the business front. But I'm bearish on the consumer front: I think it will lag behind the enterprise use case for some time, and will be seen as a social alternative to PlayStation rather than a viable alternative to dinner parties with your friends.

We seek an enrichment from our off-hours socializing that VR is a ways off from providing. But when we're working, we seek to collaborate, share, learn, and focus in quiet environments – which VR is close to providing today.

As was clear in Meta's announcement video, the wide aperture of "consumer and business and whatever else" clouds the use case and the potential. The most common criticism following the announcement was: "So... what is this again, exactly?" The change to Meta was both a commitment and a non-commitment. "We're committing to this space" without a commitment to any one thing in the space in particular. Both a bold bet and a total hedge at the same time.

This makes sense. Strategic ambiguity is necessary for consumer businesses. Who the hell knows what products consumers want? Until the Meta announcement, it felt like Facebook had all but given up figuring that out, instead waiting for the next Instagram or WhatsApp to snap up.

But business-to-business products are totally different. If you want the value-prop for Horizon Workrooms, see above. If Meta had concentrated all their energy on just this use case, far more people would have "gotten" it. There are real pain points with remote work, and if you're reading this post, chances are you got to feel them in the last couple years. And VR – and Oculus in particular – offers a real solution.

So what is it like to enter Meta's metaverse, today, for enterprise?

Well, I hope you like the "blue app," Facebook. Because you'll need it to enter Horizon Workrooms.

That's right. If you're a 30-something millennial startup founder trying to innovate how you run your remote team – e.g. me, i.e. surely their initial target persona – you need to dig up that archaic social profile you last used to check the photo tags from Sigma Chi's 2009 "Rumble in the Jungle."

_Of course_ I didn't want to connect my "real" Facebook to Oculus or Workrooms. It is a relic, and not a relic I feel any need to bring in to the workplace. Its social graph is obsolete, frozen in the early 2010s. Bringing that social graph with me into Oculus would mean turning the metaverse into a dreadful high school reunion. I'm trying to enter the future of work, not Elliott's Bar in my hometown the Wednesday before Thanksgiving.

If you're a Meta employee, I'm sure you see Facebook as more like a passport. Perhaps you know that there is no plan for data to flow between Facebook and Workrooms. It's just used to validate that you're human, that you are who you say you are.

If you're one of 7+ billion people that is not a Meta employee, you do not see it like that. Every person I've asked to connect their Facebook account to Oculus has an immediate - visceral - reaction.

This need to "validate that one is human" is precisely the kind of problem a _consumer_ social company thinks is worth solving. Zero enterprise companies are worried about that problem. Why? Because this is my meeting and I invited you to it. If my meeting invitation was sent to `shiva@acme.org` but that address actually belongs to a Russian botnet and not Acme's biz dev guy, I have bigger problems.

So I connected a fake Facebook to Oculus and then to Workrooms. And it made my experience hell. I couldn't login to Workrooms, because my Facebook didn't have enough "credibility." Defeated, I reached out to support to ask to switch in my "real" Facebook account. I was expecting to receive decent accommodations –after all, I'm trying to early-adopt the very thing the CEO just bet his $1T company on. Yet I was met with – vague threats?

```plain
AGENT (Marcelo): To unlink your Facebook Account from your Oculus Account and link it to another Facebook Account, is that what you're suggesting?

USER: Yes. Would that help me resolve the issue?

AGENT (Marcelo): And I would have to warn you that having multiple Facebook Accounts goes against Facebook Policies
AGENT (Marcelo): It may have serious consequences
AGENT (Marcelo): As having your accounts disabled
AGENT (Marcelo): Ok Anthony?
```

I know this support agent didn't mean to come off so draconian, he didn't make the rules, etc. But the fact that they have this policy, that it's so top-of-mind for their support staff – can you imagine a conversation like this _ever_ taking place with a support agent from a true enterprise company, like Microsoft?

Mark Zuckerberg is _obviously_ aware of this. The name change is as much a signal to people outside his company as those _inside_ of it. He's said explicitly he doesn't want to be Microsoft at the dawn of mobile, so fixated on Windows (Facebook) that they missed the opportunity to build their own iOS.

But requiring a Facebook account to access Horizon Workrooms was an egregious error. It's one of those errors that doesn't feel like just an oversight. It feels like a signal of the deep structural changes Meta will need to make to avoid the innovator's dilemma.

My dad isn't about to sign up for Facebook to take a meeting in VR. But he loves VR mini-golf. And when Microsoft lets his team hop from a Team chat to a VR workspace with a click, I think they're in for a pleasant surprise.

Can Meta get there before this happens?

<div style={{
  padding: "10px",
  borderRadius: "5px",
  backgroundColor: "#f0f0f0"
}}>

<b>If you've come this far, that means you're likely curious about exploring the frontier of the online office</b>. If you want to join a team on this frontier that's also building an <a href="https://sequin.io" target="_blank">insanely cool product</a>, check out <a href="https://sequin.io/careers" target="_blank">our careers page</a> and reach out to me directly: <code>anthony [at] sequin.io</code>.

</div>


Make it easy to validate your short URLs


My sister got Covid for Christmas. A few days after getting tested, she received this text:

<div style={{width: "100%", textAlign: "center"}}>
<img
src="/short-url/the-text.jpg"
alt="My sister received a text with a cryptic short URL and mention of test results"
height={700}
style={{margin: "auto"}}
/>
</div>

She receives a lot of spam and phishing texts. So, understandably, she was trepidatious: is this legitimate? Did I sign up for this?

Here's what the URL looks like (with a fake secret):

```
https://us-ca.en.express/v?c=MqWLciTcHgMlK1o5
```

The simple rules I've taught my family to counter phishing:

1. If you can, avoid following unsolicited links that are texted to you. Find another way to perform the function you're being prompted to perform.
2. If you must follow a link, always verify the domain actually belongs to who you think it does **before clicking it**.

#2 is often complicated by short URLs, like we have here.

Visiting `https://us-ca.en.express` redirects one to this page:

<div style={{width: "100%", textAlign: "center"}}>
<img
src="/short-url/ca-us.en.express.png"
alt="A google page about express covid notifications"
height={400}
style={{margin: "auto"}}
/>
</div>

So this _appears_ to be a link from a Google service. `en.express` pulls up this page:

<div style={{width: "100%", textAlign: "center"}}>
<img
src="/short-url/en.express.png"
alt="A basic page mentioning that express notifications were not found"
height={400}
style={{margin: "auto"}}
/>
</div>

On this page, there's a link to a `g.co` URL, which is in turn a link to the same page above on Google.

Of course, _anyone_ can redirect their base domain to Google. Or have a link that says "About this totally legit page" that links to Apple.

We can't trust this domain to tell us it's from Google. **Instead, we need Google – at `google.com` – to tell us this domain is from them**.

Sometimes, you can Google a domain to verify its owner. Unfortunately, that trick doesn't work for this Google domain:

<div style={{width: "100%", textAlign: "center"}}>
<img
src="/short-url/google.en.express.png"
alt="A google page about express covid notifications"
height={400}
style={{margin: "auto"}}
/>
</div>

One way to solve all this: **Both `us-ca.en.express` and `en.express` should have links to a page on google.com that explicitly mention these domains belong to Google**. That would mean a quick investigation or Google search would confirm ownership.

Still, it feels like something critical is missing for short URLs. We should use DNS or CAs to surface sibling or parent domains as a first-class concept in the browser. In that world, Google could list `en.express` as a child domain of `google.com`, making ownership easy to verify with a new browser feature.


Read from databases, write to APIs


In 2011, during the Google Plus roll out, Googler Steve Yegge [criticizes the company's struggle to build platforms](https://gist.github.com/chitchcock/1281611). In that post, he makes a contrast with Amazon. He describes Jeff Bezos' Big Mandate from ~2002. The key points of the mandate:

> _1. All teams will henceforth expose their data and functionality through service interfaces._
>
> _2. Teams must communicate with each other through these interfaces._
>
> _3. There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network._

The post went superviral. It helped kick the SOA revolution into high-gear. HTTP interfaces were stood up everywhere.

I was on a team that got swept up in the ecstasy. I guess a lot of us forgot that the architectures being described were for the two biggest internet companies on the planet.

Zooming out, there are two reasons one might justify an HTTP interface in front of their database:

1. **Security and permissions**: You want tight controls over who can see what.
2. **Abstractions**: You want to be able to change your underlying data model without affecting your consumers.

These both matter a lot when you have external consumers. And at a company like Amazon, another team - perhaps even located in another city - can feel a lot like an external consumer.

But for many teams, these goals are hardly worth the cost. The API provisioner has to build and maintain a translator from HTTP query params to SQL, one that will never come close to approximating SQL in power. And the consumer is hampered by a poor (and often poorly documented) query interface in front of the data they seek.

Does a high-growth startup have to worry about proper service abstraction? Hardly.

[Retool](http://retool.com/) is a company that understands this very well. The drag-and-drop interface for React components is compelling. But the ability to populate those components with SQL queries is the secret sauce. Building an internal REST API that powers your team's dashboards is maybe the least value-differentiated thing your engineers can work on.

That's not to say that as a startup scales, there isn't a point where the trade-off is worth it. It's just further along than most startups think.

SQL is a _lingua franca_. Reading directly with SQL queries means you eliminate a whole class of boilerplate API code, on both client and server.

For writes, on the other hand, it makes sense to go through an API. A lot of people have validation logic in their code. And writes are probably where you'll want to introduce more controls first, to prevent the team from inserting bad data or making mistakes.

Fortunately, writes are far easier to both build and use.

Hence the mantra: "Read from databases, write to APIs."

<hr style={{margin: "30px 0px"}} />

<div style={{ display: "grid", gridTemplateColumns: "auto 1fr", gridColumnGap: "20px", alignItems: "center"}}>
<img height={50} src="/sync-inc-logo.svg" alt="Sync Inc Logo" />
<p style={{fontStyle: "italics"}}>Want to help bring this architecture to every third-party platform? We're in the early days of helping developers skip API integrations with <a href="https://sequin.io">Sequin</a>. <a href="https://sequin.io/careers">Check out our job post</a> and shoot me an <a href="mailto:anthony@sequin.io">email</a>.</p>
</div>


I finally escaped Node (and you can too)


I wrote my first Node program in 2013. (It was in CoffeeScript.)

In those days, there were three factors contributing to Node's ascendancy:

The first was "JavaScript everywhere." This originally meant "JavaScript on the front-end, JavaScript on the back-end," which always struck me as a weak rationale. (This has since evolved into "might is right" and JavaScript as a _lingua franca_, which is a bit stronger.)

The second was an attraction to what Node _wasn't_. There was a growing reactionary movement in opposition to the batteries-included monolithic frameworks of Ruby on Rails and ASP.NET. This was another weak argument, as it wasn't like Ruby's Terms of Service required one to use Rails (see Stripe).

The third was by far the most important. Node had a solid concurrency story, at a time when San Francisco's dominate framework (Ruby on Rails) did not. People knew JavaScript, and callbacks provided a simpler on-ramp to concurrency than many threaded models at the time.

It was Node's concurrency that fueled its wildfire adoption. But I remember even in those early days, something felt _off_.

We had a service that would inexplicably drop hundreds of requests for seemingly random, 1-2 second periods. In my first ever Node cave-dive, I would discover the reason: an uncaught exception killed the lone process on the server. During that intermittent 1-2 second period, there was simply nobody home.

We blamed ourselves for adopting new tech too early. But I'm not sure the story has changed much in the intervening years.

I found this throwaway line from the [Deno v1 release](https://deno.land/posts/v1#http-server-performance) remarkable:

> _A hello-world Deno HTTP server does about 25k requests per second with a max latency of 1.3 milliseconds. A comparable Node program does 34k requests per second **with a rather erratic max latency between 2 and 300 milliseconds**._

I'm reminded of a lesson I learned about data structures.

**Data structures**

I once worked with a buddy of mine, Sacha, on a software project. He was _obssessed_ with data structures. We were in Saigon at the time, and packed up our laptops and traveled to the remote Cambodian island of Koh Chang. We were cut off from the world. Our beach bungalows did not have internet.

We'd spend the days working alone and meet at sunset for dinner at a nearby shack. On the first night, I wanted to talk about architecture and algorithms. All Sacha wanted to talk about was data structures.

By the second night, I had broken ground on a couple of the project's workflows. When Sacha arrived, it looked as if he'd barely slept. He told me he'd been up all night, taking long walks, brainstorming, drawing, experimenting. He had done some morning yoga, was a day into a fast -- and, finally, had some breakthroughs on _the data structures_.

I had to know: "Sacha. _Why is it always about the data structures?_" And he responded with a quote that echoed one I'd later hear attributed to Rob Pike:

> Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.

From that moment forward, I would see this everywhere. If ever I felt like my program was getting too complicated or hard to read, it was almost always a data structure problem. Since then, every time I've been humbled by another programmer's code, it hasn't been by clever tricks or algorithms. It has always been by their ingenious insight into how their program's data ought to be organized.

This principle views data structures as foundational. If you have solid foundation, the house will come with little effort. If the foundation is mud and sticks on top of a trash heap, your life as a builder is going to be complicated.

This principle applies to tools in a broader sense. You want to do the least work possible when swinging a sledgehammer, so you design it such that the hammer is a much heavier material than the handle. This gives you leverage. If you designed your sledgehammer in the inverse, you'd have to swing it harder every time you used it.

While concurrency was fundamental to Node's rapid adoption, Node's concurrency has always had foundational flaws. It's a sledgehammer, but an inverted one.

Callbacks were never optimal, and I can make that claim confidently because almost no one uses them in the greenfield anymore.

We can say the same about Promises, as async/await was designed specifically to abstract them.

But it's only a matter of time until the next floor of the house is built and async/await is abstracted. Its ergonomics are strange, even if you've grown used to them. I think a helpful perspective is the concept of [red functions and blue functions](https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/). In JavaScript, red functions (asynchronous) can call blue functions (synchronous), but not the other way around. The two also have different invocation syntax. And when one red function is introduced, it bleeds through your codebase, causing many second- and third-degree functions to turn red as well.

Async/await, and the event loop broadly, is a strange paradigm. It is tough to explain to new programmers without a lot of hand-waving. And it sounds like the exact kind of algorithmic kludge a programmer would introduce if her underlying foundation had a few flaws.

**Cognitive overhead**

Humans think through abstract concepts by mapping them onto physical analogies. My brain was evolved to understand the rough quantity and color of berries in a distant bush. It was not evolved to craft software programs. But it turns out it can perform the task (relatively) well by using the same circuitry in my brain that was intended for surviving in the wild. In my head, my program resides on a 3-dimensional plane, where functions in a file "over here" call functions in a file "over there." Reality is far from this, but the abstractions we've created, from filesystems to compilers to displays, makes this possible.

When it comes to concurrency, nothing will map quite so elegantly for my human brain than something like a _task_, be it a process in Elixir or a Goroutine in Go. The idea of a worker working on a task is just so damn easy for the human brain to reason about:

> I want to request the first five pages of this API, simultaneously. I then want to bundle the results together and deliver them to the client. So, I'll just have five of my little buddies here go and each make a request, assigning to each of them the page they are to grab -- there we are. And now, I will just sit back and wait for each of them to report back with the results.

On the other hand, for me, the idea of a callback or a promise has always required a few extra CPU percentage points. It's like a photon hitting a semi-silvered mirror: the reality of your program splits into two universes. In one trajectory, control flow continues. In the other trajectory, at an indeterminate point in the future, a callback or promise handler is executed.

Async/await is the attempt to fold the paradigm _back_ to something that's easier to reason about. It makes your program "feel" more synchronous in certain areas. But it's an imperfect abstraction and an abstraction at the wrong layer of the stack.

**Querying a database**

In Node, let's say you want to query your database in a REPL. Here's what it looks like:

```js
> const { Client } = require("pg");
undefined
> client = new Client(/* connection string */);
undefined
> client.query("select now()");
Promise { <pending> }
>
```

Something about this always felt so depressing. Rationally, I could justify it: nothing lost, nothing gained. If I wanted to board Node's async rocket to the moon, I had to accept inferior ergonomics in a situation like this. I get a promise, not a result, so I need to add additional logic to handle the promise and get a result.

And if only life were so simple:

```js
> await client.query("select now()");
REPL11:1
await client.query("select now()");
^^^^^
SyntaxError: await is only valid in async function
```

Alas, begrudged acceptance, we all got used to "the new way of doing things here."

At this moment, thanks to the proliferation of async/await, I can no longer remember the API for a `Promise` instance. So I'll just regress all the way to a callback. Which is fortunately possible, because JavaScript's "no man left behind" principle ensures callbacks will be well-supported for my grandchildren:

```js
> client.query('select now()', (err, res) => console.log(res))
undefined
```

Still no result. Five minutes of head scratching and banging ensues until I realize – for the love of von Neumann – I've forgotten to call `client.connect()`. If you don't call `client.connect()` _before_ calling `client.query()`, the `pg` client will silently push the query to an internal queue. This would be more infuriating if it wasn't so understandable – remember the flawed foundation we're building on here.

So, _finally_, I call `connect()` and then `query()` and I get a result (somewhere in there...):

```js
> client.connect()
undefined
> client.query('select now()', (err, res) => console.log(res))
Result {
  command: 'SELECT',
  rowCount: 1,
  oid: null,
  rows: [ { now: 2021-03-20T19:32:42.621Z } ],
  fields: [
    Field {
      name: 'now',
      tableID: 0,
      columnID: 0,
      dataTypeID: 1184,
      dataTypeSize: 8,
      dataTypeModifier: -1,
      format: 'text'
    }
  ],
  _parsers: [ [Function: parseDate] ],
  _types: TypeOverrides {
    _types: {
      getTypeParser: [Function: getTypeParser],
      setTypeParser: [Function: setTypeParser],
      arrayParser: [Object],
      builtins: [Object]
    },
    text: {},
    binary: {}
  },
  RowCtor: null,
  rowAsArray: false
}
```

I'll never forget, after years of only writing Node programs, the moment I first made a SQL query in Elixir's REPL, `iex`.

I start a connection to a remote Postgres database:

```elixir
iex()> opts = # connection opts
iex()> {:ok, conn} = Postgrex.start_link(opts)
{:ok, #PID<0.1330.0>}
iex()> ▊
```

I type in my query:

```elixir
iex()> Postgrex.query(conn, "select now()")▊
```

I press the Enter key:

```elixir
iex()> Postgrex.query(conn, "select now()")
▊
```

And you know what the REPL does next? _It hangs_. For a tiny fraction of a second, the beautiful little bugger just hangs. I'll never forget, it looked just like this, for a few milliseconds:

```elixir
iex()> Postgrex.query(conn, "select now()")
▊
```

I very perceptibly gasped. And then, my result:

```elixir
iex()> Postgrex.query(conn, "select now()")
{:ok,
%Postgrex.Result{
  columns: ["now"],
  command: :select,
  connection_id: 88764,
  messages: [],
  num_rows: 1,
  rows: [[~U[2021-03-20 19:40:13.378111Z]]]
}}
iex()> ▊
```

How can Elixir _get away with this_? An I/O operation like this is the precise location you _want_ async, right? Did I turn off async in my REPL somehow?? Is Elixir _not_ async???

No. Elixir can get away with this because it's built on top of Erlang/OTP. And Erlang/OTP got concurrency _so right_.

Concurrency - and the processes that support it - have been a part of OTP since the beginning. And not as a feature, but as a part of its _raison d'etre_.

When I run `Postgrex.start_link` above, the function returns to me a _pid_, which I store in the variable `conn`. A pid is an address. In this case, Postgrex has started a process that manages the connection to my Postgres database. That process is running somewhere in the background. The pid is a pointer to that process.

When I run `Postgrex.query(conn, statement)`, the first argument I pass to `query/2` is the pid of the connection process. The REPL - the process I'm typing in - sends the statement as a message to the connection process.

The human metaphor of two friends passing messages back and forth to each other applies perfectly. Importantly for me the programmer, the REPL cheerily waits until it hears back from the connection process. The call to `query/2` is a synchronous one. It's synchronous _because it can be_.

In fact, whenever an individual process does anything, it is always synchronous. On a local level, an Elixir/Erlang programmer is always thinking about synchronous, functional reductions. Including when sending and receiving messages to other processes. (And is all the while blissfully free of a red/blue function dichotomy.)

In Elixir and Erlang, concurrency doesn't happen at the function layer. It happens at the _module_ layer. You can instantiate a module as a process, and now it's running concurrently with other processes. Each process maintains its own state. And can pass messages back and forth with other processes.

What you end up with is a paradigm for concurrency that a human can easily map to and reason about. I always feel like I'm thinking about concurrency at the right layer of the stack.

In fact, for the Elixir/Erlang programmer, properly modeling your process modules _is as important as properly modeling your data structures_. I think this is why so many describe these languages as a "joy" to do concurrency with. The same rush you get when you have a breakthrough on how to model your data structure and delete 400 lines of complex business logic, you get when have a breakthrough on your process organization.

**Creation vs. Evolution**

Consider the history of Elixir: first you take Erlang, which was invented by Joe Armstrong and team to solve the extraordinary concurrency challenges that telecommunications imposes. You battle test it for many years. Then you take José Valim and team, who come in and create Elixir with the sole goal of making it more ergonomic by borrowing heavily from one of the most ergonomic languages of all time.

Like, of course you end up with something special that's a joy to use.

Languages like Elixir and Ruby are an act of creation. Ruby, for example, reduces to a creator and designer (Matz). You can feel His embrace from first contact with the language. It is welcoming. It is warm. Things feel purposeful. They make sense. Ruby's principle of least surprise makes everything feel in order.

JavaScript is exactly the opposite. JavaScript is evolution. Node is full of surprises, at every turn, for every skill level. JavaScript will [always find little ways to undermine you, to humiliate you](/bundler-wars). There is no one designer, just the cold force of natural selection. It is riddled with arcane evolutionary quirks. It is direct democracy, the people's language, for better and worse.

The history of JavaScript is complex and deeply human. And perhaps one day [the story of collapse into a singularity](https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript). I can't wait to read the book.

But might ≠ right, and I'm happy I've finally broken free.

<hr style={{margin: "30px 0px"}} />

<div style={{ display: "grid", gridTemplateColumns: "auto 1fr", gridColumnGap: "20px", alignItems: "center"}}>
<img height={50} src="/sync-inc-logo.svg" alt="Sync Inc Logo" />
<p style={{fontStyle: "italics"}}>We're in the early days of helping developers skip API integrations with <a href="https://sequin.io">Sequin</a>. Want to work on fun and challenging API and syncing infrastructure in Elixir with an amazing team? <a href="https://sequin.io/careers">Check out our job post</a> and shoot me an <a href="mailto:anthony@sequin.io">email</a>.</p>
</div>


Bundler Wars


_Looking for the dystopic Science Fiction short of the same name?_ [^1]

[^1]:
    Maybe one day we'll shoot the [Krazam](https://www.youtube.com/krazam) sketch: _Bundler Wars_ is set in dystopic 2035. Our hero Ben has fallen way behind the times (like might-use-CoffeeScript behind the times (though he won't admit it)). He's broke and desperate for work.

    After finally securing a contract, he needs to gain access to a coveted Transpiler so he can produce the final bundle. But Transpilers are expensive (after all, they do have hundreds of cores) and are tightly regulated by the Eichman (the authoritarian coalition (futilely) trying to minimize the collateral damage of the Bundler Wars on the civilian population).

    Ben's only hope is to turn to the underground - to The Unlintables.

I wrote my first React library the other day.

I avoided it for as long as I could. But when initial work was completed, I had to face the facts: I needed to setup a JavaScript bundler.

Webpack is familiar. But I scan a number of other React libraries, and none of them are using Webpack. Is that because it's unfashionable or unfit? You can never tell with JavaScript.

So I decide on Rollup. Rollup uses a plugin system. Even though my setup feels very standard, it's unclear what precise combination of plugins I'll need to get this to work. A brute-force Google-Copy-Paste-Rebundle-Refresh-Delete-Repeat loop ensues.

Eventually, I discover it's not just about the combination of plugins - plugin _order_ matters. That's because, taking a peek under the hood, it would appear that Unspeakable Things are happening. I regret peeking. Close the hood.

Here's the thing: _No one is at fault_ here.

JavaScript is everything that is beautiful and frustrating about humanity. It is the people's language, yet not one we ever democratically elected. As with much of web technology, it evolved as a series of kludges, grand plans be damned. We [tried](https://dart.dev/) [to](https://clojurescript.org/) [stop](https://coffeescript.org/) [it](https://elm-lang.org/), but this was swimming the Colorado river upstream.

Maybe we should have paddled harder.

It is only in JavaScript that we can find ourselves staring down [this massive thread](https://github.com/facebook/react/issues/13991), the issue colloquially referred to as Issue One-Three-Nine-Nine-One:

<div style={{width: "100%", textAlign: "center"}}>
<img
src="/bundler-wars/the-thread.png"
alt="The Thread"
height={300}
style={{margin: "auto"}}
/>
</div>

In my new role as React Library Author, I have become quite familiar with Issue One-Three-Nine-Nine-One. I return to this thread so frequently, I'm afraid GitHub is going to throttle me for abuse.

The issue contains ~~242~~ 287 comments (and counting). Many comments are littered desperately with emojis, like ancient cairns marking solutions as either roads to prosperity or roads to doom.

Issue One-Three-Nine-Nine-One isn't working towards a solution. It's providing a marketplace, an open Bazaar where Finders from afar can peddle their solutions - and Seekers can desperately Copy-Paste-Rebundle-Refresh - _no, that wasn't it either._

<div style={{width: "100%", textAlign: "center"}}>
<img
src="/bundler-wars/bazaar.jpg"
alt="Ancient Bazaar"
height={400}
style={{margin: "auto"}}
/>
</div>

> _Solutions for webpack! Get ye solutions for webpack!_
>
> _Here lies the command ye seek! It uses an obscure tandem of `npm-link-shared` and `prestart` hooks. Don't ask questions, yer fine!_
>
> _This 30-minute blog post will ward off Double-React Syndrome for good!_

I feel like I've tried every solution, every salve. I feel hopeless, despondent.

@apieceofbart goes so far as to (very kindly) suggest:

> "It was obviously my mistake of using two copies of React but I agree that it would be great if the error message was better."

No, @apieceofbart, this is not your fault. It's not React's fault. It's not ECMA's fault or Eich's fault.

This is our fault. JavaScript is our problem.

I get it working. How? I'm not totally sure. While my initial steps were measured, I quickly deteriorated into a debug frenzy. Copying and pasting lines from blog posts found on page 5 of a Google Search. Swapping out bundlers - just to swap them back in. `rm -f`'ing `node_modules` so many times, I was certain that those blocks of my SSD were going to go permanently dark.

My solution is far from optimal. I have a separate repo for testing my library, but I can't get `npm link` to work. But if, with each change to my library, I (1) bump the version, (2) bundle it, (3) push it to npm, (4) bump the version in my test repo, (5) `yarn install` -- my app compiles and I can test my library.

This is not sustainable. But I guess it _is_ JavaScript, after all.

I feel dejected. I have a solution, but it's unfulfilling. I feel like a cheat, a fraud, a copypasta programmer who doesn't understand the duplo blocks he's piecing together. What craftsman remains so perplexed by his tools after over a decade of experience?

A tool is supposed to _feel good_ - sturdy, predictable, precise. A tool is supposed to make its wielder _feel powerful_. JavaScript and its maddening toolchain, from beginning to end, will find little ways to undermine you. To humiliate you.

As someone that has programmed JavaScript professionally since 2013, I am convinced we are under a collective delusion. This is what Stockholm Syndrome feels like on the inside.

<hr style={{margin: "30px 0px"}} />

<div style={{ display: "grid", gridTemplateColumns: "auto 1fr", gridColumnGap: "20px", alignItems: "center"}}>
<img height={50} src="/sync-inc-logo.svg" alt="Sync Inc Logo" />
<p style={{fontStyle: "italics"}}>We're in the early days of helping developers skip API integrations with <a href="https://sequin.io">Sequin</a>. Want to work on fun and challenging API and syncing infrastructure in Elixir with an amazing team? <a href="https://sequin.io/careers">Check out our job post</a> and shoot me an <a href="mailto:anthony@sequin.io">email</a>.</p>
</div>


Drake & Future


I can imagine Drake and I having a productive conversation over a couple of beers.

I’ve run through the sequence a few times. Try as I might, the conversation takes a little while to evolve past the interview format. I nervously default to the same tactics of a first date, perhaps rambling a bit about the bar I’ve chosen or the beer we’re drinking to establish an appearance of eclecticism. Drake shrugs off the beer factoid. Not his thing. Duh. Stupid me.

In every run through, Drake starts off slumped in his chair across the table. Looking a bit tired or bored. Barely managing to patronize me with responses. He pulls out his phone often, trailing off mid-sentence to field a text. After he shoves his phone back in his pocket, he stares into the distance for a moment and then eventually remembers I’m across the table.

But we chug along. In a few rehearsals, I’ve been able to successfully tease him out. I’d plunge deep into questions about the industry, identifying areas beneath the first few well-worn layers that finally hook the guy. When I do, he smiles — he smiles! — and he leans back and says “Funny you should ask that…” and then we’ve reached a comfortable cadence. My attention can finally relax, no longer frantically jumping around, prepping the next question ahead of him finishing his answer — _is that a good question? That’s kind of a dumb question. What else have you got? Shit he’s done talking. Take a sip. Take a sip_!

Drake and I leave together, he with his private driver and I with my Uber (I naturally opted for the black car this time). And despite how awkward our fistbump-handshake-hug goodbye is, we both leave feeling it was time well spent.

I can’t even _begin_ to entertain the idea that Future and I could progress a conversation beyond the weather. I’d be clutching at every three-syllable response, digging deeper and deeper into Atlanta’s weather patterns and seasonal trends. Before five minutes pass, Future will begin wondering if I setup our meeting (at an amateurishly early hour in a strip club I’d found on Yelp) for the sole purpose of collecting information for the farm I’m constructing just outside the city.

It is clear Future and I have nothing in common. I don’t recall ever paying for bottle service or even meaningfully being a part of one. This alone seems to eliminate about half of our possible conversation topics. My idea of misusing cough syrup is taking another dose _just inside_ the recommended window. I consider my \$10 Casio a suitable accessory.

So it’s likely no surprise that my first encounter with Future’s music didn’t go so well. _Who is this guy?_ I remember the video. Then up-and-coming (now-???) producers TNGHT present Future with what insiders would call a “dope ass beat.” Future proceeds to say nothing at all on top of it. I recall him reciting lots of holidays, as if reading from a nearby wall-mounted calendar for inspiration. I believe he shook his hands in a manner that would suggest popping two bottles of champagne somewhat unenthusiastically.

I still ask this question all the time — _Who is this guy_? Forgive me for thinking the man sounds a bit uninspired at times. He opens one of his verses on _What A Time To Be Alive_ by recalling his old pals that were present on a particular night, kind of like a senile grandpa trying to set the scene for a family reunion ten years ago: “Ah yes, I know Big Meech and J-mo — they were there. And I know OG India and Suga — they were there. Oh yeah, oh yeah.”

Labeling Drake and Future’s latest effort a collaboration at times feels like a stretch. It is a stark contrast to Jay-Z and Kanye’s _Watch The Throne_. That album was brimming with expensive production. Jay-Z and Kanye tag-teamed the beats, sometimes rapidly swapping verses other times building off of each other’s material. You could tell they thought it was a Big Deal and that you should think so too.

Drake and Future spent a few days in a studio with a kid with a hot hand and just cranked out song after song until they’d exhausted his inventory.

And I can’t stop listening to it.

We don’t see two stars collaborating to create something huge. We see them just* being* huge. The album is impressive if for nothing else, underscoring the cruising altitude both Drake and Future have reached. They don’t need much to be wildly entertaining.

The album is littered with bland lines. Ghostwriting allegations should crumble before the banality of “After this club, we just change locations.” But Drake and (especially) Future’s uncanny art is breathing life into the simplest ideas. Creating an ear worm from the most basic ingredients.

Future repeats lines like “Caught the wave I ain’t surfing / Caught the wave I ain’t surfing” as if deliberately installing them in your cortex. There might be multiple layers of meaning. But just as likely, there isn’t any meaning at all.


Goodbye, friends


With the sounds of a delicate piano piece playing in the background, I sit here in the orange glow of my dimly-lit room, drinking tea, unfollowing everyone that I know on Facebook.

James ▒▒▒▒▒? Yeah, I actually don’t really care about what he’s up to.

Unfollow.

Elaine ▒▒▒▒▒? Honestly, never heard of her.

Unfollow.

Charlie ▒▒▒▒▒, a guy I haven’t talked to in three years just posted to a group I’m not a part of about a birthday party I’m not invited to and couldn’t attend even if I was?

Unfollow.

The first few unfollows are a bit uneven. Drill down to a contextual menu, click “Unfollow.” You’re asked to confirm. A flicker of doubt injects pause–maybe it would be nice to keep tabs on this guy?

Eh, unfollow.

I reach a steady cadence. My mouse seamlessly glides to the targets. I’m now comfortable enough with the procedure that I’ve also worked my left hand into the choreography. It taps the “Enter” button before my browser even finishes painting the confirmation modal.

Unfollow, unfollow, unfollow, unfollow.

But Facebook does not intend to make this easy. It begins mounting a resistance, clawing desperately back at me.

“Are you sure you want to unfollow Edward?”

Yes.

“Ok, but look at these pictures of Katie with her dog! Aren’t they coo? Awww, look at that puppy!”

Unfollow.

“Well look at Stephanie! She’s going to Disneyland! You should wish her well, like these 39 other _real_ friends of hers! C’mon!”

Unfollow.

“Fine. What about your ex? Are you telling me you don’t want to keep tabs on your ex?”

Unfollow.

“You fool. Look here. Lookie here, we’ve got Eric. Your best friend. Go ahead. Unfollow him. I dare you.”

I pause. I mean if I keep a dozen people in my newsfeed, that would be pretty manageable, right?

“Totally. Keep a few dozen people in the feed, the people that really matter to you. You need to stay in the loop. Just the couple hundred that *really *matter.”

No. I’m going all or nothing.

Unfollow.

What began as a tactical exercise to fundamentally change the way I used the most ubiquitous tool amongst my generation shifted unexpectedly into what I could really only describe as a meditative experience. An individual from my past briefly enters my consciousness as my crosshairs roll down to their latest post. Click, click, tap and they’re gone. An instance of calm as another instrument is silenced amidst the cacophony of noise.

With its command of my social domain, Facebook would probably like to drag me out into a public square and denounce me as a terrible friend and a social recluse. In fact, I would be surprised if they haven’t already spent ample resources researching subtle ways to pressure or humiliate their users. Involvement is paramount to their business. Anything they can do to keep me involved–through enforcements both positive and negative–must be explored, and the engine at their disposal is my social network.

Tonight, I feel victorious. But the war is far from over.


Posts