Xetera

🔭 Building a scalable scraper

Sat, 25 Dec 2021 00:00:00 GMT

For the past year, I’ve been working on a website for Kpop image aggregation and labeling called Kiyomi. I’m interested in imageboards like Safebooru that have an active community of fans who label every anime image to make searching easy for everyone. I love seeing how members come together to create a massive database of carefully curated content, and it’s only possible because of how much diverse content there is for the community to go through. For that reason, one of the challenges in working on Kiyomi has been to make sure that I can get fresh content to users automatically. Thankfully, there’s nothing Kpop idols love more than posting images of themselves online, so there’s an endless source of relevant material to work with.

I decided to solve this with what I know best, a scraper that I named Jiu after one of the top 7 Dreamcatcher members. In case you’re not familiar, scraping is a way of programmatically going to a website and extracting some kind of information from it. In my case, this information is image URLs and their associated metadata.

Normally this kind of image retrieval task is trivial to work with if all you need to check is a handful of pages on a single site. It’s nothing more than a

for

loop, some database lookups, and a bunch of HTTP requests. But I was planning on scraping many websites, which meant the solution had to be easily extendable to support new sites. I also had to consider another thing, which is that I don’t wanna like… be an asshole? I don’t like it when devs carelessly hammer my website with requests. So given the nature of what I’m doing, it would be pretty hypocritical of me to try to do the same for others. This meant that while working on this service, I first had to exhaust all of the “good guy” workarounds before I moved on to less-friendly measures like rotating IPs and pretending to be a real user with puppeteer.

Also, proxies that work well are expensive, and I'm cheap

This project was originally meant to be a part of Kiyomi, but I figured the problem of finding images online was generic enough that it could be

its own standalone thing

. Solving some of these problems has been incredibly fun and a huge learning process for me, so I’d like to document at least some of the ways I dealt with them.

The problem space

This isn’t crucial to understanding the problem, but I want to share the infrastructure Jiu sits on top of because it could provide important context. Also, I spent a lot of time working on this and drawing the diagram for it, so I’m going to need someone to look at it.

New images Jiu finds on supported sites are stored in a database isolated from every other service and get published to RabbitMQ for consumers to do whatever with. Jiu also runs its own webserver to allow dynamically adding and removing new providers, querying the stats of scrapes, and displaying its schedule/history.

The requirements for building this scraper are:

Creating a “plugin” system capable of supporting multiple social media sites
Turning polling into an event-based system with webhooks/AMQP
Automatically logging in to sites that require authentication
Being able to handle failure from APIs gracefully
Finding a high volume of images from different sites
Being nice and not sending too many requests to any specific site (if possible)

I am scraping respectfully

The easiest way to prevent spamming a server with requests is to just not send it many requests, duh. This is an obvious one, but it’s something that works surprisingly well for what we’re building. Because the point of Jiu is to

It’s pronounced “eventual consistency”, not “wow this program is so slow wtf?”

detect all images posted on social media and not as soon as possible. We can use this to our advantage when deciding the architecture of the scraper.

A common way of implementing these scrapers is something commonly referred to as a monitor. A monitor takes one or a handful of critical pages and bombards it with requests to detect a change as soon as possible.

This is useful for reacting to changes made in a website quickly, but it’s not that important for this specific use case. Instead, the goal here is to detect changes across

every

social media account that posts about Kpop, but in a sustainable way (aka slowly). So we want the same volume of requests, just spread out thinner over time and across significantly more pages.

Priority

Something important to recognize is that not all pages are created the same. There are accounts that consistently post good content, usually ran by companies behind groups; these should be checked frequently. There are also accounts that post infrequently, or post second-hand content (like with fan accounts); these should be checked less frequently. This requirement calls for implementing a priority system where different pages can automatically adjust their priority depending on the post frequency, on top of a manual multiplier if needed.

Originally, I went for a dynamic priority system where tasks would get re-scheduled based on how many images they found as they moved up and down on a discrete priority scale. The highest priority scrapes would get scheduled for 4 hours after a request, and the lowest would get two weeks. This worked well for changing scrape frequencies dynamically, but it created an issue with scrape distribution. It allowed multiple scheduled scrapes to fall within the same hour and get the bot rate limited by accident. This is unlikely with only a few endpoints but would become more and more likely as the app scaled, and that’s a problem.

I played around with some numbers and re-scheduling details, but the problem here was scheduling itself. I had to completely re-think how I approached it. In the end, I went with something similar to a token bucket, often used for rate limiting algorithms. Every page/account has a bucket of tokens and a rate in which it gains new tokens (called

priority

). At the start of every day, their tokens get updated by their priority. If a page has more than one token, they get scheduled one scrape per token

sometime

within the next 24 hours, and that amount gets subtracted from their tokens.

At the start of the day, a database that looks like the following will get these tasks scheduled.

Provider Type	Priority	Tokens
A	1.0	2.0
B	1.0	2.2
C	0.5	2.5

After every successful scrape, the next priority of the provider is calculated by getting the latest scrape result + the past 30 scrapes, and summing up the amount of images it found per check, weighed by how long ago the scrape was from the current day.

This makes sure that an inactive account that starts to post images gets a higher bump in priority, and an active account that stops posting images gets knocked down an equivalent amount.

When the tokens for a provider is under 1.0, it gets compounded by its priority every day even if it doesn’t get scraped. A priority can’t go below 0.07

(meaning a scrape once every 2 weeks)

and can’t go above 1.75

(3 times every 2 days)

Distribution

Once all the tasks that need to run within the next 24 hours are identified, they need to be scheduled for an exact time with an even distribution. We can’t just blindly start going through all pages that need to be sent requests immediately, otherwise we run into the exact same rate limiting problem from before.

That day’s worth of pending requests need to be grouped by their provider type —think Instagram, Twitter, etc.— and scheduled in an evenly-spaced manner using a linear interpolation between

NOW()

and

NOW() + 24 hours

/* scheduling algorithms be like */
.pending-requests {
  display: flex;
  justify-content: space-evenly;
}

This method of scheduling ensures that pages on different domains that don’t share rate limits are processed concurrently, while pages that do share a rate limit get processed sequentially.

With what we know now, let’s take another look at the scheduled tasks from last time and imagine each letter is an individual Twitter page, each scheduled to get processed at an even interval after the other.

Can you spot the inefficiency here?

We’re scheduling the same page to be scraped multiple times right after each other. We’re making sure to space the scrapes within the same provider evenly but leaving uneven spaces between repeated scrapes.

The correct order of the tasks should be more like this:

For this reason, Jiu tries its best to maximize the spacing between requests going to the same endpoint so that there’s enough time for content to update.

“Tries its best” because sadly the solution to this problem is not simple or deterministic.

Authentication

Another unfortunate problem that must be dealt with is that not every single website lets us view images without being logged in. To be able to do this, the login flow of the site needs to be reverse engineered in order to grab the necessary images. This also sometimes comes with the downside of needing to fake a user agent (or possibly other fingerprints).

Now, I know that reverse engineering and using a private API for fetching data and going outside the boundaries of what a site thinks is acceptable kind of contradicts wanting to be nice to websites and not spamming them. But using official APIs can get us rate-limited quickly, even with large gaps in requests. I’m going out of my way to be careful with the sites I’m scraping, so I deserve this. Besides, I know what’s best for Twitter more than Twitter knows what’s best for herself, and I promise I will treat her like the queen she is if she just gives me a chance.

context

Architecture and Code

For the more technical part, instead of copy-pasting functions from the codebase, I’m going to walk you through building a mini-scraper logic so you can maybe get a feel for what it would’ve been like to come up with some of the abstractions I haven’t mentioned yet on your own.

I wrote this program in Rust, but I’m going to use Typescript examples because it’s probably way more accessible for my audience. ~~Feel free to look at the Rust source if you’re interested~~ In case you’re interested, do not feel free to look through my Rust code. Please just keep reading the examples. I will take legal action against anyone I catch browsing through the repo.

Let’s start with the most fundamental idea. A requirement for this project is that we need a solution that allows fetching different forms of media from many sites. One assumption we can make is that every image/video should conform to one shared interface no matter its source. At this point, one could go for a simple class that turns a URL into an array of images.

class Provider {
  constructor(public siteName: string) {}

  async scrape(url: string): Promise<Image[]> {
    const images = await this.request(url)
    return this.extractAllImages(images)
  }
}

Try to use your imagination for the functions I haven't defined

We should now account for the possibility that this is the first time we’re gathering data from that page and haven’t seen any images. We need to add a pagination concept so our scraper can go through the page history and grab everything.

class Provider {
  async scrape(url: string): Promise<Image[]> {
    let target = url
    const images: Image[] = []
    while (true) {
      const response = await this.request(target)
      const pagination = this.getPagination(response)
      images.push(this.extractAllImages(response))
      if (pagination.hasNextPage) {
        // turns ?page=1 into ?page=2
        target = this.getNextPage(target)
      } else {
        break
      }
    }
    return images
  }
}

Now we have a way of getting all images from a website. Except, with this approach, we always paginate through the full resource for every check. We should be able to stop when we find an image we’ve seen before and only “gather” the ones we haven’t seen in a previous run. This can be solved by passing an array of previously seen images to the function to test against.

class Provider {
  async scrape(url: string, seen: string[]): Promise<Image[]> {
    let target = url
    const images: Image[] = []
    while (true) {
      const response = await this.request(target)
      const pagination = this.getPagination(response)
      const found = this.extractAllImages(response)
      const unseen = this.allUnseenImages(found, seen)
      images.push(unseen)
      if (unseen.length !== found.length) {
        // some of the images we scraped were seen before
        break
      }
      if (pagination.hasNextPage) {
        // turns ?page=1 into ?page=2
        target = this.getNextPage(target)
      } else {
        break
      }
    }
    return images
  }
}

Great, it looks like this is working fine for one provider like Twitter, but it’s still not scalable. The entire point of the project was to implement a solution that can have multiple sites plugged into it, and every provider is going to require a different way of fetching and processing data. There’s a Provider class here, but it’s not extensible in any way. We can’t configure the parts that need to be swapped out for different sites because everything gets done in a single function. Perhaps OOP is the right fix? Maybe all we need to do is extend Provider and reuse the logic somehow.

class WeverseProvider extends Provider {
  override async scrape(url: string, seen: string[]): Promise<Image[]> {
    // ???
  }
}

But what exactly goes in this function now, literally everything we had in the previous class? Aside from this.request, every single method on here is custom (mostly) per-provider and can’t just be overridden. It would be nice to have a way to almost… “template” a class method so that the if checks —like the part where we look to see if we’ve reached the end of everything we’ve scraped— and break statements stay, but everything else is customized.

As you may have guessed, there’s a way to do that. Well, not with templating, god no. That was a terrible call; what are you crazy? The issue is Provider is trying to do way too much here, and it’s getting in our way. As a rule of thumb, when you run into a problem where you need to copy-paste

logic

, creating reusable functions is usually the correct approach. However, when you run into problems where you need to copy-paste

control flow

, the solution is getting rid of tight coupling.

I mean, why does this class need to care at all about previously seen images? A provider’s scrape function only needs to scrape, and that’s it, no? It’s time to turn our single class method into a producer/consumer pair. Ideally, we need a producer that knows nothing about the outside world. And a consumer that only knows about the promise that a provider makes about what it’s capable of doing through an interface.

Let’s call the new class that will decouple that logic Scraper, and clean up the function names to reflect the new responsibilities of the two.

class Scraper {
  async scrape(provider: Provider, url: string): Promise<Image[]> {
    const seen = await this.getLatestResultsFor(provider)
    const found = await provider.process(url)
    return provider.allUnseenImages(found, seen)
  }
}

class Provider {
  async process(url: string): Promise<Image[]> {
    const response = await this.request(target)
    return this.extractAllImages(response)
  }
}

This works well for checking images. We can do something similar to re-implement pagination. Except, this time, the while loop can’t be inside the provider; it has to be inside Scraper. The problem with that is the scraper doesn’t understand what pagination means. Most sites (like Weverse) use page-based pagination, where you increase page numbers by 1 to go to the next page. But there are also sites (like Pinterest & Twitter) that use a cursor-based pagination technique, where you pass the server some opaque identifier from request one that represents the next page for request two.

Scraper shouldn’t have to know anything about how a provider’s pagination works, only that pagination is a thing that exists and needs to be passed back to a provider. Provider needs to accept and return information about pagination so it can decide what page to scrape by itself.

class Scraper {
  async scrape(provider: Provider, url: string): Promise<Image[]> {
    const seen = await this.getLatestResultsFor(provider)
    // undefined in the initial iteration
    let pagination: Pagination | undefined
    let images: Image[] = []
    while (true) {
      const result = await provider.process(url, pagination)
      const unseen = provider.allUnseenImages(found, seen)
      images.push(unseen)
      if (unseen.length !== result.images.length) {
        break
      }
      if (result.nextPage) {
        pagination = result.nextPage
      } else {
        break
      }
    }
    return images
  }
}

type Pagination =
  | { type: "cursor"; id: string }
  | { type: "page"; index: number }

type ProviderResult = { images: Image[]; nextPage?: Pagination }

class Provider {
  async process(url: string, pagination?: Pagination): Promise<ProviderResult> {
    const target = this.buildUrlWithPagination(url, pagination)
    const response = await this.request(target)
    const images = this.extractAllImages(response)
    const nextPage = this.extractNextPage(response)

    return { images, nextPage }
  }
}

This approach works perfectly! Though, the observant functional programmers

may have noticed that the thing we’re doing here looks awfully similar to

array.reduce

. We’re going through a loop and feeding the result of a function call in one iteration (in our case, pagination) back into the function called in the next iteration, much like reduce. It could be useful to re-package iteration logic with a functional helper if it makes things easier to reason about.

As a refresher, reduce (also referred to as fold in some languages) works by doing something like the following, where it takes an

A’

, an array of values, and produces

E’

as an output.

But this doesn’t quite work for us. We have an initial seed value of a URL, yes. But we don’t have an array of things we can go over. Instead we want to

create

that array of things

(image URLs)

as a result of repeatedly applying a function

(scrape)

on something

(pagination)

. Almost like a fold, but the other way around. Thankfully the thing we’re looking for here is a well-known concept like fold and is unsurprisingly called unfold.

We start with an initial value

A’

and produce two outputs, one as the value that will be “pushed” into an array and the other which will get passed as an argument into the next iteration.

class Scraper {
  scrape(provider: Provider, url: string): AsyncGenerator<Image[]> {
    // initial value is `undefined` beceause
    // the first scrape has no pagination
    return unfold(undefined, async pagination => {
      const result = await provider.process(url, pagination)
      if (result.nextPage) {
        return next(result.images, result.nextPage)
      } else {
        return stop(result.images)
      }
    })
  }
}

The hypothetical

stop

function here lets us signal the unfold to stop early if we’ve exhausted all resources. Typically, this would be done by returning an Option data type in a language like Rust, but it’s hard to represent that as clearly in JS.

This approach to iteration lets us decouple logic even further. Unlike fold which reduces an iterable into a single value synchronously, unfold generates an array of values. If the operation that is being run to generate values is asynchronous, that array can also be represented as an

infinite stream

of values that a consumer can stop generating at any point. Async iterators are great for doing this in Javascript.

const seen = await getLatestResultsFor(provider)
for await (const images of scraper.scrape(provider, url)) {
  const unseen = scraper.getUnseenImages(images, seen)
  if (unseen.length !== images.length) {
    // scraping stops here because the generator
    // is no longer being consumed
    break
  }
  await processImages(unseen)
  // delay the next iteration of the scrape
  // for 1 second
  await sleep(1000)
}

Since we’re using generators now, the scraper doesn’t need to know the conditions to look for to stop doing its job anymore; it stops when we tell it to stop inside the consumer (or runs out of pages). This means that if we wanted to, we could conveniently get the scraper to keep going even if it finds known images —like when re-fetching all images posted by a specific account— without touching the scraping logic.

Error handling and other related things can also be done now by enriching the return value of process to include more information about what happened. A provider can accept an error that it itself created in a different function and figure out how to deal with it without actually dealing with it to further decouple the process of handing off errors.

type ProcessResult =
  | { type: "success"; result: ProviderResult }
  | { type: "error"; response: Response }

type ErrorHandle =
  | { type: "tryAgain" }
  | { type: "tryAuth" }
  | { type: "cantContinue"; reason: Error }

async function iterate(pagination?: Pagination) {
  const process = await provider.process(url, pagination)
  if (process.error) {
    const decision: ErrorHandle = await provider.onError(process.response)
    switch (decision.type) {
      case "tryAgain":
        // retry logic with an unfold is nothing
        // more than recursion
        return iterate(pagination)
      case "tryAuth":
        await provider.authenticate()
        return iterate(pagination)
      case "cantContinue":
        // handle errors here :)
        return stop([])
    }
  }
  const { images, nextPage } = process.result
  if (nextPage) {
    return next(images, nextPage)
  } else {
    return stop(images)
  }
}

// inside scraper
return unfold(undefined, iterate)

Now, adding new providers is simple. It requires no duplication of control flow and not even a class. All that’s needed is some object that satisfies an interface. Notice how even with a function like onError, we are simply taking a response as input and returning

a decision

as a response, not performing an action. This framework allows the gears of Scraper and Provider to turn together without stepping over each other’s toes. Providers only know how to go, and scrapers only know how to tell providers to go.

interface Provider {
  process(url: string, pagination?: Pagination): Promise<ProviderResult>
  onError(response: Response): Promise<ErrorHandle>
  // any other functionality that could be customizable per site
}

I’ve skipped over how further implementation details of providers, persistence, publishing data, and other shenanigans work in Jiu in the interest of not boring people to death. But hopefully, this helps get a basic overview of implementing a generic scraper and decoupling business logic.

There are a couple more intriguing things I’ve come across while working on Kiyomi that I want to write about, like perceptual hashing and face recognition, but those will have to wait until I actually have any idea what I’m doing with them.

Here’s

Jiu

if you’re interested, and

Kiyomi

🐿 Golang is not a good language

Sun, 17 Oct 2021 00:00:00 GMT

A month or two ago, I put an image of a tierlist of languages that I like using in my Github readme, ranked from F to S tier. F being my least favorite, and S being my favorite. It looked something like this:

(S) Rust, Haskell
(A) Kotlin, Elixir, F#
(B) Scala, Typescript, Dart, C#
(C) Java, Ocaml, Python, Javascript, C++, Ruby
(D) PHP, Coffeescript, Powershell, Lua, Clojure
(F) Go

Part of me obviously just wanted to put that up to get reactions out of people. It’s a little bit nonsensical to try to rank programming languages in a tierlist. I’m not sure what you’d have to be looking for to compare a language like Dart with Powershell.

I had opinions on a few more languages, but the tierlist I was using didn’t have them. Maybe I’ll make my own website for ranking programming languages at some point.

Anyways, plenty of people were upset over how I placed their favorite languages in the C tier. Some people were surprised to see their choice, like Haskell and F# at the top. But

Name given to the people in the Go community

were

FURIOUS

that their favorite language was the only one at F tier and demanded an explanation from me, pressing a metaphorical knife to my throat.

So let me explain what I have against Go. Also, just as a disclaimer, I don’t like hating on languages

too much

. Most of them just want to be tools and have their own shortcomings. But a lot of problems Go experiences seem to be self-imposed restrictions through an ideology that wilfully ignores decades of important language research, which just doesn’t make sense to me.

Correctness

I don’t know about everyone else, but I am a pretty bad programmer. I often make silly mistakes in my code or skip important things in code review that I should be catching. I’m confident in my skills, but if you were to put a gun to my head and tell me my code has to work as intended in my first attempt to save my life, the last memory flashing before my eyes would be something like:

TypeError: cannot read "name" of undefined

I want a compiler to have my back. We humans suck when it comes to keeping complex conditions and details in our heads, but computers are basically designed to be good at that. I want a computer to tell me when I’m making a mistake at the time of writing the code. Hell, I wish the computer could just write the code for me to begin with. At the end of the day, I just don’t want to hope that every single person I’m working with is just a God developer who can catch every mistake before changes are pushed to production because that just doesn’t ever happen.

Golang claims to be a proponent of this concept too, and it shows that by enforcing one of the most annoying linter warnings of all time in the form of an error directly inside the compiler. The one where it prevents your program from compiling if you have an unused variable anywhere in your code.

After all, if you have unused variables, chances are you might have done something wrong. It doesn’t make a whole lot of sense to declare a variable and not use it. This is a perfectly reasonable argument to make.

Unfortunately, if you’re used to commenting things out when developing, this is going to force you to jump through a bunch of hoops and have somewhat of a clown moment in your codebase just to test stuff out.

Aside from being

unbelievably annoying

, I think workarounds like these arguably become a bigger vector for bugs in your code than leaving things unused or having linter warnings. Once you’ve decided to go down the workaround route, you’ve decided to add a hack into your code to solve this issue, but are still implicitly relying on the compiler to let you know when things are unused. When you make changes later on, and that variable turns from a simple unused variable into a legitimate bug, you feel confident in the correctness of a program that is no longer being enforced by anything in that specific instance.

Overall, I actually think this is an understandable check to have in a language that is truly concerned with correctness. If your language believes it should be as difficult as possible for a developer to add bugs in their programs, making life difficult in ways that feel unnecessary is just one of the Ls that developers have to take for extra safety. Even when that involves the language making controversial decisions or disrupting devs development cycles.

The only problem is, Go doesn’t seem to be interested in safety or correctness at all. It wholeheartedly embraces null pointer exceptions —which tends to be by far the most common source of bugs in languages that implement it— by creating an error handling system where dynamic values can be null at any time with no compiler checks involved. You simply have to remember not to create bugs and to check errors before dealing with error-able values. Quite a different philosophy from the one Go’s unused variable check would like to have you believe it adopts.

Claiming to care about safety through questionable checks and not addressing

null pointers

is kind of like if a city tried to combat crime by enacting a curfew after 8pm. Certainly a valiant effort that will help maybe… a little bit? But it makes everyone’s lives significantly harder and doesn’t do anything to address the obvious issue at hand.

func thing() {
  result, err := doThing()
  // is result nil or defined? Dunno but hope you don't forget to check
  print(result.someField)
  // runtime error:
  // panic: invalid memory address or nil pointer dereference
}

This is in contrast to a language like Rust, which won’t let you shoot yourself in the foot with any value that may be an error or undefined.

fn thing() {
  let result = do_thing();
  println!(result.some_field);
  // compile-time error:
  // error[E0609]: no field `some_field` on type `Option<Thing>`
}

Not to mention, for the amount of emphasis being put on concurrency, Go also doesn’t address the inherent problems that come with it other than by the design of the language itself (channels as a means of communication between goroutines and such). It has some checks in place to make sure that things like data races are unlikely, but doesn’t actually go out of its way like Rust to make sure they’re not possible. Safety seems to get sidelined in favor of things like easy adoptability. You don’t need your compiler to prevent the programmer from adding bugs if the programmer just doesn’t add them in the first place, right?

The lack of these kinds of features isn’t always a problem for me. Safety and static typing are really important, but not necessarily a dealbreaker for my taste. I like Elixir enough to put it in A tier (whatever that means at this point) which is a dynamically typed language, mind you. And much like Go, Elixir also does not concern itself with preventing the user from adding bugs in their code too much. But the difference is, unlike Go, Elixir embraces the fact that it can’t ensure there will never be problems. It doesn’t pretend to care about correctness to the point of adding ridiculous compiler checks in ways that don’t add any safety. Instead, it has built-in mechanisms to recover from failure as intelligently as possible as a part of the language, and decides to incorporate failure into the way problems are solved in that paradigm.

Of course, caring about safety in a language isn’t just a binary choice of “this language is safe” and “this language isn’t”. It’s just that Go’s priorities when it comes to correctness and safety seem very misguided. Most of its safety features are implemented in the form of radically opinionated language choices that make people upset like intentionally undeterministic map iteration or the lack of support for variance in its type system. Decisions that make a good-faith effort in trying to prevent the user from firing a bullet at their feet with a gun that does a concerningly good job shooting at feet by design.

Simplicity

The reasoning behind this archaic error checking method and the unused variable thing is that Go wants to be as simple as possible. One way to ensure that simplicity is by only allowing a single “Go way” of doing things. Every engineer working on a shared codebase should be able to hit a problem with a Go-shaped hammer and get a predictable Go-shaped solution out of it. If you’re not allowed to handle errors in any way other than always explicitly passing it back to the caller, for example, there’s no debate to be had about how error handling should be done. Except for when there is a debate, of course.

Go’s philosophy seems to hint that complexity only comes in the form of complex abstractions. That if you are not given complex tools like union types, generics, or macros, your programs will necessarily be simpler. And this to me seems like a child’s way of thinking about how complexity arises in programs. Simplicity does not exist on a linear scale of

complicated

and

not complicated

languages. It’s a delicate balance across many things in multiple dimensions. Sure, having too many tools and too many ways of doing the same thing is going to cause complexity like it does with a language like C++, but so will having no tools to deal with complex issues. After all, you can’t be expected to chop down a tree with a steak knife. When your go-to abstraction for generalizing solutions is copy paste, that extra effort you had to spend learning and understanding new abstractions is now going to be used to maintain code written with a steak knife; making understanding a solution built with simple ideas much more difficult than it has to be.

You’d much rather have access to more complex concepts to represent more complex problems in those situations, because complexity isn’t inherently bad. Maybe you don’t want to have to understand what the hell a

Semigroup

is to add 2 lists together, but you also don’t want to have to copy paste this meme

if err != nil {
  return nil, err
}

13 times in the same function and then copy paste that function N times for all data types only to solve an otherwise-trivial problem.

if _, _, err := c1.WriteMsgUnix(wb, nil, a2); err != nil {
  t.Fatal(err)
}
if _, _, _, _, err := c2.ReadMsgUnix(rb2, nil); err != nil {
  t.Fatal(err)
}
if _, err := c2.WriteToUnix(wb, a1); err != nil {
  t.Fatal(err)
}
if _, _, err := c1.ReadFromUnix(rb1); err != nil {
  t.Fatal(err)
}
if _, err := c3.WriteToUnix(wb, a1); err != nil {
  t.Fatal(err)
}
if _, _, err := c1.ReadFromUnix(rb1); err != nil {
  t.Fatal(err)
}
if _, err := c2.WriteToUnix(wb, a3); err != nil {
  t.Fatal(err)
}
if _, _, err := c3.ReadFromUnix(rb3); err != nil {
  t.Fatal(err)
}

Fun little example from Golang's own standard library. Sure, it's a test file, but I think that only proves my point of needing situational complexity even more.

This idea of simplicity-gone-too-far appears in many places. Go makes extensive use of “tuples” when returning 2 or more values from functions, but tuples don’t actually exist in the language. It uses a keyword range to iterate over maps and arrays, but has no concept of an iterable data type. Plenty of features are implemented in the form of very specific compiler magic, and the only explanation for it is that it’s to prevent you from building your own abstractions. If a language is opinionated in not adding complex abstractions, allowing the user to build their own would kind of defeat the whole idea.

Community

Simplicity on its own is not necessarily a bad thing to strive for, even when done well. But the way Go approaches it is a little… well, I’ll just let you decide how you feel about this quote from Rob Pike, one of the creators of the language.

I find it really funny how he's talking about devs at Google (a position that many developers can only dream of having) as being too inexperienced to handle a good language.

Go doesn’t only want to make your life easier by preventing arguments about menial things. Its philosophy is based on the idea that you’re incapable of dealing with the tools you think you should have. Why don’t we have generics in the language? Because generics are hard and you just graduated from college last year. Copy and paste it instead.

Maybe I’m just being weird here but I don’t really want to use a language built by people who think they’re better than me…? Like, don’t get me wrong, almost everyone working on Go certainly is. I just don’t think this is a healthy way of building a relationship with a community, though this might also be the brash nature of Rob Pike in specific.

This seems to be an issue that shows up in the Go community fairly often. If you look at discussions of problems and proposals like proposal: leave if err != nil alone? #32825, they tend to have a predictable format across the board. In one corner we usually have the “please sir I just want to use a good language” team, asking for changes that they feel are going to make their lives easier. And in the other corner, we have the “you just don’t understand the language man” team, trying to convince the users that they don’t actually want the things that they want; alienating the userbase with a mix of patronization or stockholm-syndrome depending on where it’s coming from.

"Have you considered not having a problem with this?"

It seems like this divide exists because Go was never built for a community in the first place; it was built for Googlers. It only ended up taking off with the rest of the programming community due to being open source and its amazing approach to concurrency, which was the main problem they were trying to solve at the time. But because it was never built for a community, it has all of this baggage that makes sense in the context of a company imposing restrictions on its employees with what they believe to be the correct approach, but not with a language open for everyone to use.

It took the language maintainers a ton of pushback from the community on their puritanical ideals about what they think the perfect language should be like to create a language that the people actually enjoy using. Sadly, they’ve tucked all of that away in the promise of a Go 2 that’s still yet to come.

There are some honorable mentions that I didn’t include here like Go’s

hilariously bad

approach to date formatting.

and a couple others, but I didn’t think they deserve the same criticism because they’re just weird corners of the language and don’t make as much of an impact on the overall experience.

Overall, Go is not built by inexperienced people or anything, or even a bad language depending on what you value. It makes a lot of opinionated choices and deliberate compromises of important ideas to solve problems that I just don’t consider to be worthwhile at all and sometimes even harmful. Go has not learned anything from decades of programming language research and re-implements all the same problems that language designers have expressed endless regret over. The concurrency is awesome, but the pain of actually using the rest of the language simply isn’t worth it.

Just use Rust instead.

🔑 Building a robust permissions system in TypeScript

Sun, 05 Sep 2021 00:00:00 GMT

Have you ever tried to deal with permissions in your app and run into this issue?

function onSubmit() {
  if (!user.hasPermission(Permission.WRITE_COMMENT)) {
    sendMessage({
      message: "Sorry you can't post comments!",
    })
  }
  submitComment(user, form.comment)
}

AGH! You just forgot to return early from the permission check and ended up trying to write a comment anyways! If you made this mistake on the frontend, you probably got some annoying errors from a backend that hopefully didn’t make the same mistake you did. If you did it on the backend, you just introduced a pretty serious vulnerability in your app. Let’s hope this got caught in code review. Otherwise, you probably caught these hands instead.

How do we prevent this? A bug like this seems impossible to avoid other than by just being very careful; we would need TypeScript to read our minds here if we wanted it to prevent us from doing this, wouldn’t we? Err… well, no, but that might be a planned feature for an upcoming TypeScript update.

We’re making an assumption here, where we believe that all users passed into writeComment are going to have the WRITE_COMMENT permission. The problem is, we’re not telling the typechecker that there’s any difference between an authenticated user who can write comments and a user who’s logged out, so it can’t fail when we use an unauthorized user to do authorized stuff. We’re doing a check with an if statement to gain valuable information about the properties of the data we’re working with, but we’re not turning that information into something the typechecker can use to do its job; the type of user stays the same despite our explicit check. It’s an anti-feature to use a single User type when we’re working with two incompatible ideas, so why not try to come up with a better system that’s capable of representing the data we’re working with more accurately to make sure bugs like this can’t even compile.

We’re going to start by defining our base User, which will represent any user on the site who may or may not be authenticated.

type User = {
  // this is the only relevant property for our system
  permissions: number
  // with any other user-related property
  // you can think of
  name: string
  age: number
}

This specific example uses a bitfield to store the permissions a user can have, which is represented with number. What you use for storing permissions internally is not important. Unless you’re working on the Discord API and want to provide your users a bad experience, you most likely won’t be directly exposing things like bitfields to the consumers of your API anyways. This concept works with any permission system.

Now, let’s define a new AuthorizedUser type to declare a user who’s had a permission check, as we’re going to need a distinction between the two if we want to allow the typechecker to do work for us.

type AuthorizedUser<T extends Partial<Permissions>> = User

Whoa hold up. Why is there a T generic here if it’s not even being used? Shouldn’t you get rid of it?

This type parameter is known as a

Phantom Type

in Haskell (and probably is a functional programming concept in general). Phantom types are parameters that are not a part of the type definition and are only used to differentiate one type from another based on some criteria to provide additional safety.

The idea is that an AuthorizedUser<T> should always be a valid User but not the other way around. Authorized users are defined with User, making them compatible (or easily convertible in the case of other languages). But User is distinct from AuthorizedUser<T>, so it cannot be used interchangeably with it. Phantom types help us fine-tune this subtyping relationship using the constraint of permissions.

Unfortunately for us, TypeScript does not care that AuthorizedUser has a different name from User because it uses a structural type system. The name you give a type is not taken into account by the typechecker when comparing two types, only the definition itself. If the fields of any two types match, TS considers those two to be compatible. So we’re actually forced to use T in the definition in order to distinguish it from User. You could argue this makes it not a phantom type anymore, but I could argue that you’re dumb and win the argument immediately.

type AuthorizedUser<T extends Partial<Permissions>> = User & {
  // just here to make typescript happy
  __permission__: T
}

This will work, but we can accidentally access __permission__ even though it’s not meant to be used, because it won’t ever be assigned a value. We can use a

unique symbol

to make sure it’s not accessible in our code by anything other than that symbol, which we will not be exposing to the rest of our codebase.

declare const phantom: unique symbol
type AuthorizedUser<T extends Partial<Permissions>> = User & {
  // just here to make typescript happy
  [phantom]: T
}

It’s worth noting that if you have any properties that should only exist for an authenticated user, you could declare them in this type. There’s even a way to make them only appear for a specific type of permission, but that’s a story for another day.

Now that we’ve declared users, it’s time to head over to the original submitComment function definition.

async function submitComment(user: User, comment: Comment) {
  // your favorite commenting implementation here
}

Let’s change this function by baking the assumptions we’re making about the authorization level of user in our mind directly into the type signature.

async function submitComment(
  user: AuthorizedUser<WriteComment>,
  comment: Comment
) {
  // exact same code as above
}

Remember, nothing changes with the implementation of this function when we change the user type, since an AuthorizedUser<T> is a valid supertype of User. The responsibility of disambiguating between these types is on the consumer, not the provider. As far as we’re concerned inside submitComment, we’re still working with a regular User.

If you're working in a team, using a declarative approach like this also makes the capabilities of this function and your intentions crystal clear to your coworkers. What is a restriction for typecheckers is often documentation for humans. 😊

Let’s take a look at the implementation of Permissions and WriteComment.

const writeComment = { writeComment: true } as const
type WriteComment = typeof writeComment

const readComment = { readComment: true } as const
type ReadComment = typeof readComment
type Permissions = WriteComment & ReadComment // & anything else

Huh, this seems like a strange way to do permissions, why not use an
enum
the way God intended?

Using enums is certainly clearer and easier to read, but it comes with a pretty severe limitation that makes it unusable for our purpose. Consider the following code:

enum Perm {
  WriteComment,
  ReadComment,
  BanUser,
}

function renderCommentsSection(
  user: AuthorizedUser<Perm.WriteComment | Perm.ReadComment>
) {
  // implementation here
}

For this component to be rendered on your app, the user would need to be able to either write comments OR read them. If they can’t do either then there’s no component to even display. Nothing particularly wrong about this case. Enums are just numbers or strings at the end of the day, and this is something that you can easily represent with an enum; it’s just 0 | 1. But what about this case?

function renderCommentsModerationPanel(
  user: AuthorizedUser<Perm.ReadComment & Perm.BanUser>
) {
  // implementation here
}

This is supposed to be a component that renders a control panel for moderators that have the ReadComment AND BanUser permissions but uhhh… what on earth would be the type of 1 & 2? You can have a number that is either a 0 or a 1… but what is a number that is a 0 AND 1, or 1 AND 2? That just doesn’t make any sense. Actually, there

a type for 0 & 2 and it’s called never, not very useful because it just doesn’t exist. You can’t have one number be two different numbers at the same time.

For us to be able to implement permissions, the permissions need to be represented by a type that can be both unioned

AND

intersected by each other. Sadly that rules out all primitive types including enums if we’re interested in representing more complex permission restrictions. The good news is objects fit this criteria, even if they do feel a little bit more jank than the alternatives.

Now we need a function that allows us to create a new user type and assign it to a new variable to use as an authorized user.

type AuthorizeResult<T extends Permissions> =
  | { type: "ok"; user: AuthorizedUser<T> }
  | { type: "fail"; reason: string }

function authorize<T extends Permissions>(
  user: User,
  permission: T
): AuthorizeResult<T> {
  // imagine an actual function implementation here
  if (someCondition) {
    return { type: "ok", user: user as AuthorizedUser<T> }
  } else {
    return { type: "fail", reason: "User does not stan loona" }
  }
}

There’s a lot going on here, so let’s dissect it piece by piece. We’re defining an authorize function that takes in a regular User and permissions we want to check for, but it’s not returning an AuthorizedUser. The user might not have the correct authorization level, so we’re telling the compiler that the result might either be a successfully authorized user, or a failure with a reason explaining why. The typechecker

WILL NOT

allow us to access the result of this function without explicitly checking to make sure we successfully got an authorized user out of this.

So the bug we had earlier turns into this with our new system.

function onSubmit() {
  const result = authorize(user, writeComment)
  if (result.type === "fail") {
    sendMessage({
      message: "Sorry you can't write comments!",
    })
    // agh! we forgot to return again
  }
  submitComment(result.user, form.comment)
}

But instead of crashing in runtime, this time we get a compile-time error as soon as we make the mistake.

Property 'user' does not exist on type 'PermissionResult<WRITE_COMMENT>'.

We didn’t return from the if statement and the TypeScript compiler couldn’t narrow the type down to an AuthorizedUser<WRITE_COMMENT>, as soon as we add the missing return it works again. 🎉 We did it!

This still feels a little bit weird, though. Assigning a new user variable feels unnatural, and this doesn’t look like our original check either. That’s ok, TypeScript gives us a way to narrow the types of variables conditionally using something called

Type Guards

Let’s create a type guard for checking permissions.

function hasPermission<T extends SomePermissions>(
  user: User,
  permission: T
): user is AuthorizedUser<T> {
  return Permission.authorize(user, permission).type === "ok"
}

How does the bug look now with the type guard?

function onSubmit() {
  // user has type User here
  if (!hasPermission(user, writeComment)) {
    // user has type User here
    sendMessage({
      message: "Sorry you can't write comments!",
    })
  }
  // user has type User here
  submitComment(user, form.comment)
}

And we still get an error, nice!

Type 'User' is not assignable to type 'AuthorizedUser<WRITE_COMMENT>'

When we return from the if statement properly, we get the same sweet TS typechecker magic.

function onSubmit() {
  // user has type User here
  if (!hasPermission(user, writeComment)) {
    // user has type User here
    sendMessage({
      message: "Sorry you can't write comments!",
    })
    return
  }
  // user has type AuthorizedUser<WRITE_COMMENT> here
  submitComment(result.user, form.comment)
}

It can infer that we’ve narrowed the type of User to the correct AuthorizedUser<T> by performing the check. 👏

This system we set up can work for more complex situations and is capable of dealing with just about any kind of authorization-check-related bug, not just an early return.

Here’s a playground link of a complete implementation if you want to try this out with an editor that displays the correct compiler errors https://tsplay.dev/WK8Zzw.

What we just did here was using a side of static typing that most people don’t even know exists. Yes, static typing can be used to distinguish data apart from each other, but it can also be used to separate

incompatible ideas

with the same underlying data into different types.

Your job as a programmer should not be to check types in code review. You suck at it; that’s the typechecker’s job. Make it do as much of the work for you as possible because it never makes mistakes. Communicating with your compiler is almost as important as communicating with the people you write the code with.

Gotchas

There are a few caveats to this system, of course. Anyone can cast a User into an AuthorizedUser to circumvent compiler checks. But checking for the presence of unsafe casts and functions with the wrong permission requirements is a much simpler mental task to do in a code review than having to remember to check every permission-related bug every time permissions are involved.

The goal of this is to be able to constrain the slightly unsafe code to just a few functions that are extensively tested, vs being scattered everywhere around a codebase that gets updated with every new commit.

😠 Misleading educational videos

Sat, 28 Aug 2021 00:00:00 GMT

I love the internet as a resource for learning new things. I love YouTube as an easily accessible, exciting source of information for beginners. I have a long article describing just how much of a fan I am of online learning, but it comes with its own set of problems.

Teaching is not simple; communicating ideas takes a lot of very intentional effort to pull off successfully. It’s easy for people who don’t have a full grasp of a topic or find it difficult to convey what they do know well to unintentionally spread misleading —or sometimes straight-up incorrect— information. In my experience, most channels that spend time creating online content do their due diligence and make sure what they’re putting out is high quality and correct, but this is not always the case.

I want to specifically talk about one video today, which I think is a pretty egregious example of what I mean.

What’s wrong with this video

Now, right off the bat, the title… I don’t know. It’s a bit weird, but I guess you gotta hustle and get views, right? I don’t like it, but it doesn’t change the content; it’s just a red flag that makes this video stand out to me as potentially low quality. It’s just ever so slightly sus.

As a quick summary, the person in the video is going through an OWASP Juice Shop setup and demonstrating an injection vulnerability with user creation. It allows him to modify the payload in the user create endpoint to include properties that get sent directly to the database without any validation. This kind of thing can happen on a backend that looks something like this.

app.post("/user", async (req, res) => {
  // no checks, send everything straight into the database baby
  const result = await db.createUser(req.body)
  res.json(result)
})

Now, this is a pretty serious vulnerability, but it’s not realistic for big websites. It can happen with an inexperienced developer using a cringe backend like Express.js where there’s no validation built into the webserver and zero type-safety, but it’s not something you’re going to be running into in the wild very often. Injection is literally #1 in the OWASP list of top 10 web vulnerabilities, so it’s clearly a big problem. However, simple injection exploits like these are not going to be found in the first place you’d be looking for them, like a sign-up form. You probably don’t make mistakes calculating something like 20 + 10 if you’re double-checking and have people going over your work as well.

The way Loi finds out this vulnerability (I hope that’s his name, I haven’t watched this YouTube channel before) isn’t really discussed, and he tries to sell the process he’s going through as something that works for any website; which is obviously not true, but we can suspend our disbelief in the name of clickbait magic for now. In the video, he points out that the backend is storing sensitive information in the JSON Web Token (JWT) issued to the user and goes through it to find potentially juicy information.

This isn’t an incorrect observation since there’s a password and totpSecret field in there, which are definitely scary-sounding fields I don’t want to see in my client. But the critical thing to note here is that this layout of the JWT with all the unexpected “sensitive” content lets an attacker infer that there might be a missing sanitization step between fetching user data and creating a JWT. The JWT claims might represent the exact structure of what a user row in the database might look like since a carefully crafted token would probably not contain these fields. The data stored in the JWT is

NOT

a vulnerability or a security problem by itself, as these fields should only be accessible by the user the token is being issued to. It’s just a strong clue for an attacker that gives insight into what the backend might be doing wrong.

The step of inspecting the JWT gave Loi an idea of what the database schema of the site looks like, and it let him make a good guess for how he could change a payload that might be lacking validation from its inputs; the same way it lacked validation for its outputs. It’s not actually a part of the vulnerability he ends up exploiting, and yet, he keeps going on about storing sensitive information in a JWT the entire time and never once mentions injection, which is the real culprit here.

Why?

He later goes to the user registration page, pulls up burp suite, and talks about how the JWT thing is making it possible for him to create a new user with premium, which makes no sense. The roles in the JWT are not insecure or sensitive. It’s not uncommon to store user roles in a JWT as it’s signed, and a user can’t change it. If there was proper validation being done, the JWT would’ve made zero difference, and the exploit wouldn’t be possible. These two things are not in the least bit related to each other. This exploit is still possible with or without a JWT storing information that is not even sensitive to begin with.

The only problems that arise from storing user roles in tokens are logistical, and not security-related. For example, if you store roles in a token, you have to reissue the user a new token when you want to change their roles which can be kind of a pain to deal with if you don't have a flexible setup for that sort of thing.

It’s also possible to set claims on a token without straight-up dumping an entire DB row into it and giving attackers insight into your DB schema and your app’s security flaws.

This was clearly something that confused viewers, seen by comments like these from people who convinced themselves that they just became experts on stateless authentication after watching a video of an exploit that had nothing to do with JWTs.

I understand that these types of videos are meant to wow beginners and maybe get them excited about security, which is awesome! But the way this video is presented seems like it’s meant more for experienced devs, with the way it’s filled with a lot of technical jargon that would not be very useful for a beginner. It almost feels like it’s an attempt to impress inexperienced people with a bunch of complicated technical stuff by someone who forgot that there are people who understand what he’s talking about watching the video. I don’t like this at all. This person is clearly experienced in this field and is someone that plenty of people look up to as an expert, yet he’s not doing a good job communicating his knowledge clearly.

I’m all for showing cool stuff to people to get them interested in a field. But there are plenty of beginners who are still learning about these concepts who consume the content people like this guy create as a foundation for their knowledge, and that has some serious implications in situations like these. There’s a way to combine being engaging and being correct, but it’s not always easy. If you’re willing to compromise this much on the latter part, I don’t think you should be creating educational content, or at the very least, content that leads viewers to believe they just learned something new from an expert.

Btw, huge shoutout to the one person in the comments who actually understood what was going on lmao.

🏫 Why does school suck so much?

Sat, 21 Aug 2021 10:07:10 GMT

There’s a quote that every breathing human ever has either heard of before or recited for themselves that goes:

Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime.

And I agree with the premise, but I think this is a pretty outdated saying and I’d like to propose a third alternative that is inspired by the words of the famous philosopher Ron Swanson.

Don’t teach a man to fish. He’s a grown man; he can learn to fish on his own. Instead, show him why he might be interested in it over something else. Show him why you’re passionate about fishing. Take him out for a boat ride, and get him excited about how good it feels when you catch something after patiently waiting for hours. Describe to him the details of fishing that make it feel so rewarding to sit with a pole in your hand for a whole day. But

please

, for the love of God, don’t teach him how to fish. He doesn’t need you for that.

Admittedly, this version doesn’t quite roll off the tongue, so I’m probably not going to be remembered for it when I die, but I think it touches on an important idea that I want to discuss.

Why we teach

For a very long time, most of teaching was something we had to do out of necessity. If you wanted to survive, you had to learn how to hunt, how to avoid predators, how to gather supplies for your tribe; someone was equally as obligated to teach those things to you. You learned what you had to learn, so you could do what you need to do. Sure, even in the early days, people were concerned with arts and other non-essential skills, but they were not much more than side projects. Something you did when you were free from the work someone or something forced you into doing. If little Ugubwe wanted to eat that night, he had to hunt with the men of the tribe, and then he could learn to paint on the cave walls from the tribe elders later if he wanted to.

As of fairly recently —specifically during the time of the industrial revolution— we learned to channel this idea into institutions capable of growing a large population of little people into skilled factory workers years down the line. This wasn’t so much based on necessity; we realized that like… I don’t know man like this learning thing might be a good idea I don’t know… So we went ahead and set up schools to actually be usable by the general public and came up with the

incredible idea

that being born into a family that has multiple servants should not be a pre-requisite to learning how to read. You still didn’t get to paint on the cave walls on your own terms though. You learned the absolute basics of literacy and whatever skills you needed to become a good, obedient factory worker.

This way of learning worked well for its intended purpose for many

thousands

of years, but because it was motivated by necessity or covering the absolute basics, you were never asked about what it was that you wanted to learn. You weren’t nudged into discovering a passion for yourself. You were either given a task you had to do for a lifetime or had to find out what that was if you wanted to survive.

Now in

2023

, as our global economies are transitioning out of the

In the

three-sector model

, the secondary sector refers to an economy that is based around the conversion of raw material to products through factories and processing.

and into the

In the

three-sector model

, the tertiary sector and above refer to an economy that is primarily focused around the sales and marketing of goods produced in the secondary sector, as well as the creation and sharing of information.

, we’re finding that there is objectively a more diverse pool of work to be done compared to what our parents and grandparents had to do. Statistically speaking, if you’re in middle school today, by the time you’re ready to join the workforce, you’re most likely going to be working a job that doesn’t even exist yet. When it comes to career choice, we can now choose from a plethora of options, and we live in an era where all the information we need to learn new things is at our fingertips. Yet, for some reason, the fisherman is still trying to teach us to fish even though we have the option to do anything else.

What if I don’t want to become a fisherman?

Remember when you were a kid and you heard that question? You know the one I’m talking about…

So Jakey, what do you want to be when you grow up?

Every kid’s nightmare. Of the million things you could choose as a profession, chances are you had no idea what you wanted to be when you grew up. What were you even interested in?

I'm assuming that the people who'd be interested in reading my blog were kids when video games were a thing.

? Art? Chatting with your friends about random nonsense on Skype late on a school night until your dad finally walked into your room at 3 AM and confiscated your phone while muttering something under his breath about how irresponsible you are? Were you ever given an opportunity where you could discover new things you liked that fell outside your comfort zone of interests as a kid? Probably not. We have so little time to explore ourselves, the world, and our interests while going through school. Yet, for some reason, we assume that kids will have some idea about what they’re going to be doing for a living after high school or college when we never bother to allow them to discover their passions. This phenomenon did not use to be relevant in the past, but I would like to argue it certainly is now.

Hope you didn’t forget about your homework of figuring out what you want to do in life. You had 12 years to hand it in!

When we were still hunter and gatherers, at least after you learned to hunt, you had all the skills necessary to become a hunter. How does this translate to school now? Do students end up developing an interest in the fields they spend a long time learning? Do all kids want to become English teachers and biologists? Do these courses prepare them for the future where they might have an opportunity to work in a field they actually enjoy? Oh, I’m sorry, were you sleeping in your

investment banking

class? Maybe you should pay more attention in your

forensic investigation

class next time because not only do you not have any of the skills necessary to work in these fields; you probably don’t even know what it means to do those things as a career.

The bottom line is that school does absolutely nothing to expose you to different fields you could be interested in exploring. Even though educational institutions are capable of helping you explore various interests with an educator who could help guide you into discovering a passion, that’s rarely ever the case. You will learn the basics of a few subjects some boomers decided you should know many years ago —with a few revisions here and there— and you will be released into the wild with no valuable knowledge to further your career or your desire to learn.

Also, I don’t mean valuable knowledge in the same way an annoying middle-schooler would talk about it in science class going:

I’m referring to the kind of information YOU consider to be valuable. Whether that’s the knowledge that will help you succeed in life, like writing a good resume, or knowledge you find fun and engaging just for the sake of it. If you want to find something you’re interested in pursuing in the long term and you think school is going to help you with that, I have some bad news for you.

All of this happens because there’s one important thing that’s changed in the last few decades that educational institutions have not managed to keep up with, which is that information is now easily accessible online. The benefits of “giving” someone knowledge aren’t as pronounced when you can simply learn those things yourself with little effort. Combined with the increase of the “surface area” of information you need to know, it’s clear that students need guidance and mentorship to navigate their way to what will work well for them, not knowledge. You can learn things from anywhere; that’s not the issue anymore.

How we enjoy being taught

The reason I feel so strongly about this topic is that I know most people —if not all— are interested in learning and are curious by nature. School does a terrible job at bringing out this incredible quality in us, but it’s there. One of the best places to witness this is in the comments section of educational math and science videos on YouTube, which many people still label as

procrastination

and

not real learning

to this day.

You can see the amazement in people as they realize that learning is an incredible feeling and that school doesn’t have to be synonymous with education. Unfortunately, that is the way most of us feel, as we get put through this system starting as early as age 5, and it leads to a painful internal conflict for kids who realize the current education system isn’t their cup of tea and they don’t perform well in school because of it. They often think that they are just too dumb to understand concepts like math and science and arrive at the inevitable conclusion that they don’t like

learning

Obviously this isn’t specific to YouTube, but there’s clearly something about this style of learning and exposure of new concepts that touches people’s hearts to the point where those who don’t see themselves as an average school fan find themselves to be an average YouTube enjoyer. So what is it about this medium that feels so different from school? Let’s look at that question by exploring the differences between what it means to be educator in the school system vs online media.

Educational Differences

YouTube forces content creators to compete for your attention at any given moment. If the video you’re looking at is not enticing enough, chances are you’re just going to click on the next recommended video. Content creators win customers over by getting them to subscribe, but they still need to work hard to make sure that the videos that they’re putting out continue to be engaging for their audience. As a viewer, you have the opportunity to explore a tremendous amount of alternative content with little work on your end. You can also choose not to watch YouTube anymore, at which point the content creator still loses a customer. There’s no single point where a creator gets to relax and stop caring about how engaging their audience thinks their work is because the customer lifecycle can break at any moment.

Teaching online is also difficult work, and naturally, the people who choose to go through with it are those who have a special knack for teaching and content creation. I can only think of a single one of my college professors who could even imagine of doing what they do at school on the internet where you have to actually be good at explaining ideas to see any amount of success.

Shoutout to my man Henry Carnie, one of the only college professors I had who was actually good at what he does and got me interested in History.

Things go a little differently for formal education.

First, society expects you to attend school.

Don’t believe me?

What’s the kind of reaction do you think you get after you tell a family member that you don’t want to go to university?

You need a reason to choose not to go to school, and a good one at that too. That’s an

incredible

leverage for educational institutions. For that reason, schools have to do relatively little to ensure that new students keep coming in year after year in terms of enticing students other than existing in a convenient physical location and not having a bad reputation.

They also win over customers when students enroll, and that’s generally where the effort they have to put in ends. Schools don’t have to care about how engaging you feel the education is because they have no reason to. An enrolled student will continue attending unless the school does something catastrophic, and when that does happen, it’s usually not an education-related issue either.

Wait, what’s that? You’re saying that you always have the power to choose to attend a better school as a customer/student?

Uhuh

, which school are you going to move to, the one that’s the second closest to your house? Gosh, I hope the education style there is better than the previous, even though that school has no reason to care either. Fine, you could choose to attend an online school, so you’re not restricted by the physical location, but how are you supposed to tell apart an online school with a teaching style that resonates with you? There’s no metric for this kind of thing. Schools get classified by prestige and cost, and we all know prestige is not an indicator of whether or not a school sucks to attend by any means.

There’s an industry standard set by traditional schools where, as a student of the school you attend, your opinions and satisfaction of these institutions are nowhere to be found in the list of things that influence their decision making. The schools make themselves look good doing the exact same thing they’ve done for the past 200 years, just with different curriculums. Your aunts and uncles, along with the rest of society, continue to pressure you to attend them year after year.

Your mom also went to UCI sweetie; you should apply there too.

This stark contrast between the two types of education goes a long way to explain why students have been desperate for an alternative but haven’t been able to push for any meaningful change to get what

they

want to be prioritized. I believe that as remote learning starts to become more common and the importance of having a degree begins to decrease, there’s going to be a major shift in how we do education at the undergraduate level and below. Students, for the first time in a long while, are going to start looking for an alternative way of learning that satisfies their own needs instead of someone else’s.

I can assure you that schools have done less to drive interest in civil engineering on a per-capita basis than this YouTube channel has. These viewers and educators are the ones who will be leading the slow transformation of the education standards around the world in the next 50 years.

As some final thoughts, if you’re still in school, I highly encourage you to do your best to explore different fields you might be interested in by yourself instead of waiting for school to do it for you. One of the most liberating feelings in the world is knowing that you can learn anything you need without being dependent on someone teaching you the curriculum. When you have the freedom to explore the things

you

like, learning no longer becomes a chore.

If you know you like eating fish, don’t wait for someone to give you a fish or teach you fishing. Just Google it and learn it by yourself.

Try to take what I've written here with a grain of salt. I never finished my degree and I'm obviously projecting my own experiences in the education system. I'm also heavily biased towards a way of learning that works for me and isn't compatible with the traditional school education system.

I recognize that this isn't the case for everyone out there and that everyone has their way of learning that works with or without a formal education system. I'm only interested in helping people find out what that style is for themselves and not necessarily pushing my preferred method onto anyone else.

🤖 Bot Taxonomy

Sat, 21 Aug 2021 00:00:00 GMT

Bots and automation have been something I’ve been fascinated with ever since I started programming. I got into it by looking into Discord bots and quickly learned about things like scraping and other kinds of automation to add new functionality to different tools I was working on at the time. I eventually ended up getting into building scalpers to pay for my college tuition, which was an interesting experience but not something I enjoyed doing (like college itself). Today, I still work in a field that involves botting, just on the other side of the table.

There are many different kinds of bots whose tasks range from crawling the internet to make websites accessible for everyone to making businesses lose

as much money as possible

. The story of bots on the internet is one made up of protagonists, antagonists, and countless supporting characters; villains who try to pillage metaphorical cities for gold and wreckless heroes who put everyone else at risk in order to save the day.

So, how common are bots on the internet? Well, it’s hard to tell, but it seems that:

62% of all web traffic comes from bots

I’ll admit, that’s a slightly dubious number. There are other figures from different journals that are way off from this one but seeing as there are so many bots out there, I figured I’d like to categorize them and the rich ecosystem they exist in to see what they’re all about.

I will be using an extremely scientifically rigorous model created by the brightest philosophers and

Botany

Noun

The scientific study of bots.

This is real btw don't look this one up.

of our generation to explain where I place different aspects of the automation ecosystem in (my) moral system.

?

😇 Lawful Good

The good guys.

Any automation that is beneficial to you and conforms to how you expect it to behave with pure intentions. These are generally your standard search engines. They’re the reason why the internet works in the first place. Aside from a few ridiculous crawlers like Baidubot, they will go out of their way to ensure your site isn’t affected by what they’re doing.

They’re also the reason why it can be challenging to protect yourself against unwanted automation. Sometimes a good solution for blocking bad bots you don’t want might need to be reconsidered if it also blocks the “good bots” which would ultimately be a net negative for your site.

😇 Neutral Good

The

Blue team in computer security refers to the party that is on the defending side of a system that is being attacked.

These companies/services will ensure that

Neutral Evil

does not manage to automate your website and cause harm. They fight for good, but the methods that they use to provide their services result in privacy violations for real users through the use of things like browser fingerprinting and aggressive data collection.

Protection like Google’s ReCaptcha or Binance’s weird puzzle slider thingy also often make it difficult and sometimes impossible for a lot of users who rely on accessibility tools like screen readers to continue using the site.

Bots are very hard to distinguish from users who use the same technologies for legitimate purposes. If you want to make things harder for bots, you inevitably make things harder for accessibility tools, and any bypass you add to help these users will be abused by bots who pretend to be legitimate users requiring accessibility. This often puts fighters in this category in a very tricky position of lose-lose where there’s just no good solution to the problem that doesn’t involve major compromises.

Neutral Good

is in a constant battle with

Neutral Evil

as well as regular users who feel like they've become collateral damage in the name of web security and have come to expect it to work like magic without involving any drawbacks.

😇 Chaotic Good

The kinds of automation that are ultimately for the benefit of everyone but have to be exercised with caution.

One of the best examples of this is probably

Shodan.io

. They are constantly scanning your infrastructure and exposing the tools running on it like databases, SSH access, FTP servers, and more. If you use it properly, it’s a tool that can be used to detect unwanted exposure, but it’s also a tool that allows bad actors to uncover information about your infrastructure that could’ve otherwise stayed hidden.

It’s not a good practice to rely on

The practice of trying to secure access to important data by making things confusing and relying on the incompetency of a 3rd party rather than actually securing the access required for it.

For example: hiding the key to your house under your doormat, hoping nobody finds it, instead of keeping it somewhere that's actually secure.

, but there’s definitely an argument to be made for how Shodan can be abused for the same reasons it can be a useful protection tool.

🙄 Lawful Neutral

Overall useful utilities with no specific benevolent or malevolent purpose.

IoT devices in specific have no specific alignment other than doing what they need to be doing, which is generally positive. They don’t provide a service in general that can be considered inherently malicious or benevolent.

🙄 True Neutral

Mostly uninvolved 3rd parties.

These people aren’t really motivated in any particular way. They just want to go about their day without having to deal with a bunch of stuff. This is usually people who are just trying to create something new for the sake of learning or for fun, and often get caught in the crossfire between abuse and protection.

🙄 Chaotic Neutral

The services who provide ammunition for everyone else.

There’s a huge part of the bot ecosystem built around vendors that sell different kinds of tooling that can be purchased to execute abuse at a bigger scale. The most common examples of these are services like

Luminati

(I guess they call themselves Bright Data now) or

Oxylabs

which will sell you proxies that “lend” you their IP address which can be used to bypass anti-abuse measures like IP bans.

There are other ones like

2Captcha

where you can pay a child in Bangladesh 3 cents to click on fire hydrants in captchas for you so you can deploy an army of automated bots to get past human checks. This is why captchas are borderline useless on sites targeted by botters who have the money to pay for these kinds of 3rd party captcha services.

Similar to selling guns on the black market to unknown buyers,

Chaotic Neutral

doesn’t technically carry out any of the abuse themselves, instead enables another party to do it in exchange for money. The morality here is questionable at best, and they’re well aware of what their target market is.

😈 Lawful Evil

I don’t actually know what fits into this category.

Any technology or service that is malicious enough to cause problems is motivated by money or something equally as selfish. I can’t think of any sort of destructive automation in the wild that claims to be self-righteous to justify its actions. The first thing that comes to mind is something like

Anonymous

, but they don’t fit into the category of automation.

😈 Neutral Evil

The

The red team (in contrast to the blue team) is used to refer to the attacking side in computer security.

People using automation in this category do not have malicious intentions or want to cause harm to the services they target. However, they believe that they have the right to obtain what they want without any regard for rules and do not care about whether or not their actions will harm the other party involved.

This is the category of bots that cause the price of GPUs to skyrocket, get your Roblox account hacked, purchase the limited edition Supreme crowbar you wanted, and add millions of fake views to videos on YouTube.

The industry behind these bots has grown to such a scale that most people have started considering it a fact of life. Speaking of sneakerbots and scalpers in specific, they’re sold as a service to users who want to run their own bots to buy limited edition stuff and have a rich support system backing up their customers. They’re run by dedicated developers who charge a subscription fee and always stay up to date with the latest changes to sites to make sure users of the bot can continue to bypass new protections.

For people at the top tiers of this category, it’s not a matter of whether you’ll be able to stop them, but for how long. When the person abusing your site makes a million dollars a year offering their services, there’s not a whole lot you can throw their way to tire them out.

Neutral Evil

generally believes that they're not doing anything wrong; instead, they think they're simply hustling and that it's up to everyone else to step up their game instead of being forced to stop.

Despite not being motivated by malice in specific, they’re ready to go to similar extents as

Chaotic Evil

to get what they want.

😈 Chaotic Evil

The bad guys.

These are the actors whose actions center around bringing harm to the services they target. Whether it’s caused by revenge or something else, the automation in this field primarily serves to inflict damage to the victim without an obvious direct benefit to the actor. This mostly includes things like botnet DDoS or destructionware.

Surprisingly, there’s not a whole lot that falls into this category. Unlike anime supervillains, most people in real life are not motivated by pure evil; there’s almost always an underlying motivation that explains people’s actions beyond just being evil and wanting world domination.

I’m a huge fan of this little world we’ve built around automation and the endless cat and mouse game between the two parties. Hopefully, I’ll have the opportunity to work more in-depth in this field and learn more about different kinds of ways to keep raising the bar of competency needed from bot developers to abuse services.

I've never actually played dungeons & dragons so if the reasoning behind where I put things in my alignment chart seems off then uh... I don't really care, sorry nerds.

❄️ Blizzard's Downfall

Sun, 01 Aug 2021 00:00:00 GMT

In 2017 I moved to California to live with my mom and go to one of the nearby community colleges to study computer science. One of the things I couldn’t stop thinking about when I got there was the fact that my mom lived 10 minutes away from the Blizzard Headquarters in Irvine.

It felt surreal, but when I checked the map for the first time, it was there in all of its glory, no cap. All fax no printer. The company that makes the game I’ve been obsessed with for years as a kid, standing a very exhausting jogging distance away from the bed I go to sleep on every day.

Because I had only just gotten into programming at the time, I couldn’t even fathom the idea of trying to get in contact with someone working there or applying for a position/internship. So for the next three years, I went past the Blizzard HQ every day on my way to school, wishing they’d just give me a chance.

At the end of 2019, I finally managed to build up a somewhat decent resume of projects and got the courage to apply for an internship. I had a lot of the skills that were required in the listing so I felt like I had a chance at the time.

A couple of weeks after I applied, I went for lunch by myself at one of the Japanese restaurants near their HQ, and I saw like 10 Blizzard employees with badges walk in for their lunch break which I thought was

incredibly

cool and freaked out for a bit. After what felt like an hour of contemplation, I finally got the courage to say hi to a girl who was waiting for her food at the table next to me. We talked for a little while, and I found out that she works on Overwatch as a character design person and —unsurprisingly— enjoys what she does. She also wished me luck when I told her I applied for an internship and that I was waiting for a response. I had to cut it short after this because I was obviously bothering her, and she just wanted me to stop talking. Or at least that’s what my brain kept telling me, as it always does whenever I want to chat with strangers.

To put this into perspective, I have

really

bad social anxiety when it comes to interacting with people IRL so this was one of the few instances where I managed to get myself to talk to another person without being forced to in my 3-4 years being in the US.

But of course, as with every big company, they get many applications; hundreds of people get rejected every time, and I was one of them. They also took over

6 months

to get back to me, which I thought was funny.

The rejection hurt at the time. I had stopped playing games and World of Warcraft for a while, so it’s not like I wanted to work on WoW in specific, but this was my dream since coming to the US.

The gaming industry

One of the reasons why it was harder for me is because Blizzard is no ordinary company. I wasn’t competing against people who wanted a job to make money; I was competing against people whose dream was also to work at Blizzard.

This is one of the incredible benefits of being a game company. There are so many people out there who’s only dream is to share the incredible experiences they’ve had playing games with the rest of the world, and enable others to feel the same kind of joy they did playing games as a kid (or still do as adults in many cases). I don’t know a whole lot of people who find the same kind of purpose in working on health insurance software.

Because of this, the people who are currently working at big game companies like Blizzard and Riot Games are mostly interested in keeping their positions working on the games they love before anything else. They’re significantly less likely to complain about the routine unpaid overtime. They’re not interested in raising hell over injustice. They’re prepared to take on a sea of L’s because a lot of them have conditioned themselves to believe that they’ve been blessed by the miracle of working at an amazing game company like Blizzard and that they have to shut up and take the good parts with the bad because that’s just how things are.

Despite this disincentive to speak up, Blizzard has finally found themselves buried neck-deep in lawsuits over mistreatment. They’ve managed to drive a group of people who will suck it up and take whatever is thrown at them to the edge to finally say something.

Blizzard employees staging a walkout on the same street I used to drive on going to school every day.

Imagine how bad it must have been for it to get to this point. Of course, if you’ve been following along with the gaming industry, either as a gamer or a game developer yourself, you don’t have to imagine anything. You already knew that this was something that’s been going on for years and probably had to tell yourself that

it’s just how things are

to be able to turn a blind eye to everything going on and to keep enjoying the games you love playing. Gaming is supposed to be chill and not like other jobs. So what if the company COO is farting on your coworkers’ faces and touching them inappropriately? So what if your company recruiters are asking female security experts if they like being penetrated at conferences? They’re just like… having fun man, maybe women need to stop being so angry all the time, right?

As much as it pains me to say this, the reason why Blizzard might finally be facing the consequences now isn’t because they’ve fostered a hostile work environment for women for years by hiring bigoted, misogynistic men. It’s because they hired

stupid

bigoted and misogynistic men who are not only bad at their job but bad at being a bigot too.

Here’s a quote from NPR about the blizzard scandal.

One woman said her manager told her she wouldn’t be promoted because “she might get pregnant and like being a mom too much.”

Now, I’m not a manager myself, so I don’t know the details of these kinds of things inside out, but I’m pretty sure you cannot say something like this. Not just morally, but as a part of your job, you cannot say these kinds of things to your employees because you’re becoming a liability and (little did he know) opening the company up for a lawsuit. There are plenty of male managers out there in the gaming industry who have a relentless hatred for women who manage to keep their positions because they know what they can and can’t say as a part of their job. They’re shitty people, but they are somewhat decent managers and get to live off the “isn’t it so great to do the thing you love for a living” debuff women get hit with dealing with them every day.

In Blizzard’s case, it seems like the people in charge still think they’re in a college frat house and don’t have any idea what they’re doing, even when it comes to their job. You can imagine incompetency of this degree probably contributed significantly to the decline of World of Warcraft. It seems like your skills at drinking games in the office and harassing women is a more important factor in your value at Blizzard than the kind of work you do.

In most cases, intolerant people of all types learn that they need to keep quiet about their views early on. They understand what is socially acceptable and what isn’t and will only open up about their views to others once they realize that the other person is equally as intolerant about the same things.

Managers at Blizzard don’t seem to understand this basic idea either. And it’s only because of this gross incompetency that we’re finally hearing something about mistreatment from the public and following along with a lawsuit. Who knows how much longer they could’ve kept this going if they simply played into the industry standard of sexual harassment without crossing into the

blatant

sexual harassment category.

Looking back now, I have to thank the university relations team at Blizzard for turning down my internship application because it allowed me to start my position at Top.gg in the summer of 2020.

You cannot have a positive work environment that is rooted in institutional gender discrimination, harassment, and a lack of accountability. You also cannot have a positive work environment where your privileged superiors see you as an amorphous blob of an employee and don’t think you should be allowed to talk about problems that affect your life when you’re at work. People tend to be their best selves in environments where they feel respected and don’t have to worry about being treated differently because of who they are because you

don’t

stop being a unique individual when you sign a contract.

🚫 India banned my site... I think?

Sat, 03 Jul 2021 00:00:00 GMT

A few days ago I got a ping from one of the moderators in the Top.gg Discord server about something I’ve never seen happen before.

I’m sorry? Why would a documentation site get banned by the Indian government? Surely this is a mistake. Anyways, best to check it out to see what’s going on.

Ok, this is a strange situation but the google docs ban seems like a fairly reasonable lead to follow.

After a little bit of digging around on Google I found out that Google sites were never blocked to begin with.

In an order dated 23 June 2014, the Delhi High Court upon a request made by Sony Entertainment ordered 472 file sharing and file hosting websites blocked, including The Pirate Bay, Google Docs, Google Videos, and Google’s URL shorterner (goo.gl) […] However, it was reported on 7 July 2014 that an updated court order blocks just 219 sites. Included are many file storage and torrent websites, but no Google sites.

So I guess that’s not it then. Time to start looking somewhere else. Should I just be getting in touch with the Department of Telecommunications at this point? I know plenty of Indian developers in different communities I hang out in, maybe they can help me figure out what’s going on.

I’m also not convinced this is a global ban. Sure, a documentation site doesn’t receive that much traffic but we have a lot of Indian users and someone would’ve either let us know about this ban a long time ago, or there would be more than one person with the same problem, right? It seems very odd that I’m only now hearing about this and only from a single user.

Let’s check in with people in a different server.

That would be nice but I don’t think they want me, or really anyone emailing them, ever.

At least I can definitely relate to not wanting emails.

Oh they do have their twitter page up there though. Seems a bit silly but I know my mom managed to figure out some problems with Cox in the US only after DMing them on Twitter so maybe that’ll work for me here too.

Please ignore the fact that I yoinked the docs page from the

apollo docs

I know I don’t have a ton of experience sliding into DMs but I was really hoping the government of India of all people wouldn’t leave me on read.

They also have an option to leave feedback on their site. I could try sending the same message there.

Welp, that went about as well as I thought it would. I figured this might’ve been because of my non-indian IP but sadly it didn’t work with a VPN either. I dug around in dev tools and it’s something about my captcha being invalid…? I can’t even be bothered tbh.

It’s 10 AM at this point and I don’t exactly feel like making an international call inquiring the status of my national ban. Also, I hate phone calls, please don’t call me. It might be a good idea to check if anyone else is having problems reaching the site. That should help narrow things down a bit.

Oh so it’s a DNS issue then. Looks like it’s banned at the DNS level for Indian citizens. But windows user (if he doesn’t unknowingly have a custom DNS setup) is able to reach the site through his ISP’s DNS. Maybe different Indian ISPs enforce censorship at different levels?

I know this is something Turkish ISPs do extremely inconsistently as well. Plenty of VPN sites like NordVPN are banned by ISPs that have close ties to the state but others don’t seem to be bothered to enforce certain bans. At least I haven’t had any problems connecting to VPNs with the god-awful network provider I’m currently using, and yet my parents have. I could totally see this being the case for India as well.

Great, so we have two people using Jio, one of whom can view the site and another person who can’t. Could the ban be enforced geographically? That’s not unheard of but it feels like the explanations we’re reaching for here are starting to get more far-fetched every time.

At this point, I take a break from the intense

investigative journalism

to go back to doing what I was supposed to be doing for half an hour or so.

I come back and have the idea of looking at the traceroute for the URL to see what IP the docs URL is being resolved to when I’m on an Indian VPN. But to my surprise I see…

WHAT???

Why is it working now? Maybe it’s just a VPN thing?

Why do these things always happen to me?

I hate web development

To be fair, it’s possible (and most likely the real reason in this specific case) that it works now because the DNS query got cached at the router and/or browser level when Sky switched his DNS server to

1.1.1.1

and got a successful response. But I’ve seen the same thing happen to at least 3 people where they inexplicably regain access to the site after a while, even when they haven’t switched their DNS server.

Possible explanation

What I think is going on here is that some Indian ISPs (most likely Jio in this case) are using some sort of heuristic to determine whether or not a DNS query to a site should be blocked, and then they are caching the result in their own DNS servers to speed up subsequent lookups.

This is completely normal practice for DNS servers, but unfortunately it feels like they might be committing the cardinal sin of caching. Where they leave out part of the context that changes the response per-user, and are instead using a single shared cache key for every user when doing any lookup. On the user side of things, this is equivalent to opening up Instagram and seeing the feed of a different person every 30 minutes.

Chances are, the DNS query was cached when the original person who tried to reach the docs page got an error. Any subsequent requests going to the same DNS server ended up returning the same cached error response for as long as the TTL lasted. Then when the cache got reset, it got replaced with the correct IP originating from the request of a non-blocked user. Which allowed the original blocked user to also be able to access the site.

As for the reason for the censor itself, I’m still not sure but my team and I are speculating that it’s caused by an old censorship of GitHub. Our docs page was being hosted on GitHub Pages and the ban is mostly likely the result of a lingering censorship of the GitHub Pages IPs. We’ve since moved to Netlify and it seems that solved the issue. Or at the very least I haven’t seen anyone bring this up since then.

Solution

If you’re being affected by this, the easiest solution is to change your DNS server your device or preferably your router uses to

1.1.1.1

or any other public DNS server that isn’t limited by Indian censorship. You do not need a VPN as it doesn’t seem like the DoT uses packet inspection and relies only on DNS lookups. If you’re using GitHub Pages to host your site, switch over to an alternative like Cloudflare Pages or Netlify to make sure you don’t run into the ~~great~~ mediocre firewall of India.

📚 Truthiness is not loose equality

Sun, 13 Jun 2021 00:00:00 GMT

There are a lot of beginners out there and even experienced developers who are confused about what it means for something to be truthy in Javascript. There’s also an assumption that many people never go out of their way to validate, where they think loose equality (

) is the same thing as truthiness.

Truthiness makes some level of intuitive sense to most people who deal with it. If you show someone the following code snippet, chances are they will have a vague understanding of what’s going on. Even if they aren’t Javascript developers.

if ("test") {
  console.log("This will run")
}

if ("") {
  console.log("This won't run!")
}

But the explanation that is given is often muddled with misconceptions about how comparisons and equality works in Javascript.

Truthiness

It might be a good idea to take a step back and establish what it means for something to be truthy first. Many people get a feeling for how this works but don’t actually dive into the details. In simple terms, truthiness is just what you get when you throw the variable in question into the

Boolean()

function.

Boolean("") // false
Boolean("memes") // true

so any time you’re calling an if statement, what that if statement is actually doing is this:

if (Boolean(x)) {
}

The

The standard that defines how Javascript is supposed to behave.

defines a function that it refers to as ToBoolean, and all implicit boolean conversions including the explicit Boolean() behave according to it with the following lookup table.

Argument Type	Result
Undefined	Return false.
Null	Return false.
Boolean	Return argument.
Number	If argument is +0𝔽, -0𝔽, or NaN, return false; otherwise return true.
String	If argument is the empty String (its length is 0), return false; otherwise return true.
Symbol	Return true.
BigInt	If argument is 0ℤ, return false; otherwise return true.
Object	Return true.

This might come as a surprise to some. What is truthy and what is falsy is clearly defined for each data type and has nothing to do with equality. This is the reason why you might expect arrays inside if statements to behave the way they do.

if ([]) {
  console.log("This is always true")
}

An array is an object which is always true regardless of its contents. If you’re a python developer, this is likely confusing since python’s lists implement

bool

differently and empty lists are considered falsy just like empty strings.

So if truthiness isn’t based on ==, what’s going on with loose equality?

Loose Equality

Javascript has this quality where it tries to make guesses for you about what you’re trying to do and change the data it receives in order to have things make sense. This is called coercion and it isn’t necessarily always bad. The examples above are a fairly common way for Javascript devs to do if statements. Checking things like

if (array.length) {
  console.log("This array has a length of at least 1")
}

to make sure an array isn’t empty. The same coercion happens when using the loose comparison operator ==, so why is plopping things inside if statements acceptable but using == isn’t?

For one, ToBoolean is a unary function, meaning it has a single input and so it’s a lot easier to wrap one’s head around than == which is a binary operator with 2 inputs. But another reason is because loose comparison is actually a recursive operator…

That’s right, loose comparison is implemented recursively. Meaning that comparison between two non-matching data types could recurse as much as 4 times, converting the inputs in each iteration into a different one and re-running the equality check.

There’s an extremely useful website on this topic that visually shows exactly how this process works between different kinds of inputs.

But basically in summary, equality is defined separately from truthiness. The only similarity they share is that truthiness is a form of coercion and loose equality can make use of a similar type of coercion although equality itself does not rely on the concept of truthiness.

🖼️ Optimizing images with WebP in Node.js

Sun, 04 Apr 2021 00:00:00 GMT

If you’re going to be accepting and serving images in your webserver, it’s a good idea to make sure they don’t take up HUGE amounts of space. Storage doesn’t grow on trees you know. A portion of what makes images take up a lot of space is its quality, images that look better take more space than those that look 💩.

File formats are another portion of that puzzle. Not all formats were created the same, something like GIF is an incredibly wasteful format as it stores each individual frame separately whereas something like MP4 stores the transitions between frames which makes it vastly more efficient for representing animation.

WebP is one of those file formats that allow for lossy compression in a way that (on average) makes high resolution images look better than what JPEG provides.

😔 Virgin PNG

7.1MB

That's crazy. Nobody wants to download this file and you don't want to serve it either.

😎 Chad WebP

433KB

Now this is the good stuff. Look at how much smaller that is. Imagine not using webp in 2023.

Can you see the difference? I sure can’t. Try opening them in a new tab and zoom in, it’s still not noticeable. The conversion from Png WebP here is saving us that many bytes which makes the image load faster. That’s a

The example image is optimized to load faster on your browser. These values are from the raw images that were used.

for free.

There are some implications of using improved file formats of course. A browser first has to be able to support the file format that you’re using so outdated browsers like Internet Explorer and Safari don’t support it fully.

Getting Started

Let’s get started with converting images in Javascript by installing the basic dependencies. Make sure you use the npm version of these commands if you’re not on yarn.

$ yarn add fluent-ffmpeg

We’re going to be using ffmpeg to do our image conversions. If you’re not familiar with ffmpeg, it’s the god tool for working with any kind of media. Images, sound, videos, gifs, subtitles. It’ll take anything and modify it in any way you want. Here’s a simple example that works on the commandline.

$ ffmpeg -i image.png image.jpeg

Nothing too fancy here, we’re simply setting image.png as the input file and turning it into a image.jpeg. Ffmpeg is able to infer that we want a

JPEG and JPG refer to the exact same file type, just with different names.

output We can also pass in any kind of option we want. If we’re converting to WebP, we can control the quality of the resulting file through these options.

$ ffmpeg -i image.png -quality 50 image.webp

This will produce a fairly low quality but an amazingly small webp version of our image. The default ffmpeg value for webp images is 75 which I find to be adequate but you can feel free to play around with it as you like.

CLI programs can be a bit of a pain to run so let’s jump on fluent-ffmpeg which is a more convenient version of the api.

import ffmpeg from "fluent-ffmpeg"
import * as fs from "fs"
import { Readable } from "stream"

function convertImage(imagePath: string, outputName: string) {
  ffmpeg().input(image).saveToFile(outputName)
}

convertImage("./image.png", "./image.webp")

This produces the same output as the one above, but it’s not very practical, let’s look at an express example where a user uploads an image using multipart/form-data.

import express from "express"
import multer from "multer"
import fs from "fs/promises"
import { convertImage } from "./lib/image"

const upload = multer({ dest: "uploads/" })

const app = express()

app.post("/profile", upload.single("avatar"), async (req, res, next) => {
  convertImage(req.file.path, "./image.webp")
  // image.webp is now available on disk
})

That’s it, that’s all you need to convert images to webp, wasn’t so problematic was it? Except there’s a problem, this method of doing conversions is extremely inefficient. We’re writing the original file to disk first even though we just end up reading it again right afterwards. Thankfully multer allows only saving things in memory without writing to disk with the following.

const upload = multer({ storage: multer.memoryStorage() })

There are basically no other multipart parsers that do this. I spent many hours looking and multer was the only one. Kinda crazy...

Images, which are just binary data in memory, are represented with the Buffer data type and ffmpeg by default takes a file path. Being a CLI program, we can pipe a stream of bytes into it in the CLI like this.

$ cat image.png | ffmpeg -i pipe:0 image.webp

In english terms that’s equivalent to

Take the contents of image.png, pipe it across this program which takes its input from pipe:0 and convert it to image.webp.

pipe:0 here being stdout that we pushed the contents of image.png into. If you’re not familiar with unix pipes definitely check look into them.

In fluent-ffmpeg, doing this kind of piping requires the input to be a ReadableStream. The conversion between Buffers and Streams is a bit hacky and it involves manually disabling some internal methods but it’s doable and works just fine.

import { Readable } from "stream"

function convertImage(image: Readable, outputName: string) {
  ffmpeg().input(image).saveToFile(outputName)
}

export function bufferToStream(buffer: Buffer): Readable {
  const readable = new Readable()
  readable._read = () => {}
  readable.push(buffer)
  readable.push(null)
  return readable
}

this now lets us patch the previous function like so.

import express from "express"
import multer from "multer"
import { bufferToStream, convertImage } from "./lib/image"

const upload = multer({ storage: multer.memoryStorage() })

const app = express()

app.post("/profile", upload.single("avatar"), async (req, res, next) => {
  const stream = bufferToStream(req.file.buffer)
  convertImage(stream, "./image.webp")
  // image.webp is now available on disk
})

This method works pretty well but we still have the same issue on the other end. We’re saving an image to disk but often we want to do something like upload it to S3 or save it somewhere else and writing the image to a file ends up being a pointless operation that takes time and resources. For this example we’re going to pretend we want an output we can pass along to the AWS sdk to upload on S3 instead of saving on disk. https://wasabi.com is a great service for this.

Thankfully, ffmpeg allows outputting a stream just as it allows for a stream input. To do that we will use a PassThrough stream which, as the name states, just passes the stream through. We use this to capture the output of the conversion.

import { Readable, PassThrough } from "stream"

function convertImage(image: Readable, outputFormat: string) {
  const passthrough = new PassThrough()
  ffmpeg()
    .input(image)
    .outputFormat(outputFormat)
    .stream(passthrough, { end: true })
  return passthrough
}

Now that we’re passing a stream to ffmpeg it can no longer infer what filetype it needs to turn the image into so we need to explicitly tell it that we’re looking to convert to a specific format.

For testing purposes, we can try writing this to a file manually.

app.post("/profile", upload.single("avatar"), (req, res, next) => {
  const stream = bufferToStream(req.file.buffer)
  const output = convertImage(stream, "webp")
  output.pipe(fs.createWriteStream("./image.webp"))
})

Aaand it work- wait… what the hell is this?

The file is corrupted?? What on Earth could be going on?

Debugging stream issues

The exact same command we tried worked with a file, why is it not working with a stream as opposed to a file target? To start debugging this, we first need to understand how the webp spec works and how ffmpeg interacts with streams.

The webp spec says that images need to have the block size of the file at the beginning of the file like so.

A working webp image will have a file size signature where the image size is between the first 4th and 7th bytes.

However, when we pipe the data directly out of ffmpeg without writing to disk using this command (which is the equivalent of the original problematic js code).

$ cat image.png | ffmpeg -i pipe:0 -f webp pipe:1 > output.webp

The exact same file looks like this… why?

Essentially, ffmpeg doesn’t know the size of the image as it’s going over it so it uses 4 null bytes 00 00 00 00 as a placeholder and plans to seek back to the beginning of the image when it’s done to replace it with the size of the image it just read. Except when piping to stdout, ffmpeg is unable to seek back. Try to see if you can spot where the missing 2e 8a 03 00 bytes might have gone instead of the header.

I’m not sure why, but instead of warning you that piping a webp output to stdout will not work, ffmpeg decides to go ahead and just do it anyways and adds the file size at the end of the image. This will likely be fixed in the future by failing with a warning, but as of version 4.2.4, ffmpeg does weird things with webp outputs. (The same thing applies for other file formats that have block sizes in the header like FLAC files).

One interesting caveat to this is if the image is small enough for ffmpeg to read in one go without separating into multiple chunks (less than a handful KBs), it doesn’t have to seek back to the beginning and will properly include the file size in the header from the start.

Putting it Together

This bug isn’t the end of the world for us though. We can fix it by simply moving the last 4 bytes of the image into the 4-7 byte position. Javascript has very efficient methods on typed arrays and buffers that let us move data around without much overhead called copyWithin and slice. copyWithin is similar to the performant memcpy and memmove in C (it even has the same gross type signature) and slice returns a view without copying anything much like the &str type in Rust. Definitely not at all confusing with Array.prototype.slice which is often used to explicitly copy arrays. Good job Javascript.

One problem we run into when implementing this fix is that streams are not directly editable and we have to consume the entire stream first and then concat the chunks into a single buffer before we can start picking apart the bytes.

import { Readable, PassThrough } from "stream"

function convertImage(image: Readable, outputFormat: string) {
  return new Promise((resolve, reject) => {
    const chunks: Buffer[] = []
    const passthrough = new PassThrough()
    ffmpeg()
      .input(image)
      .outputFormat(outputFormat)
      .on("error", reject)
      .stream(passthrough, { end: true })
    passthrough.on("data", data => chunks.push(data))
    passthrough.on("error", reject)
    passthrough.on("end", () => {
      const originalImage = Buffer.concat(chunks)
      const editedImage = originalImage
        // copy everything after the last 4 bytes into the 4th position
        .copyWithin(4, -4)
        // trim off the extra last 4 bytes ffmpeg added
        .slice(0, -4)
      return resolve(editedImage)
    })
  })
}

And it worked!

Now we can publish to s3 easily with

import express from "express"
import multer from "multer"
import { S3, Endpoint } from "aws-sdk"
import { credentials } from "./config"
import { bufferToStream, convertImage } from "./lib/image"

const upload = multer({ storage: multer.memoryStorage() })
const app = express()

export const wasabi = new S3({
  endpoint: new Endpoint("s3.wasabisys.com"),
  credentials,
  s3ForcePathStyle: true,
})

app.post("/profile", upload.single("avatar"), async (req, res, next) => {
  const stream = bufferToStream(req.file.buffer)
  const output = await convertImage(stream, "webp")
  await wasabi
    .putObject({
      Bucket: process.env.MY_BUCKET,
      ContentType: "image/webp",
      Key: "test.webp",
      Data: output,
    })
    .promise()
})

And that’s all! You have an optimized webp optimized image uploader for your app. You might want to also convert to some fallback formats like jpeg if you want to support older browsers just in case.

☄️ Integrating Typescript with Next.js

Sat, 02 Jan 2021 00:00:00 GMT

Info

This post is specifically aimed at Prisma users but you should be able to use it with any ORM (or no ORM ot all if you roll that way).

One of the most annoying things about integrating different technologies together is losing static typing along the way. This is my biggest complaint when it comes to working with databases without an ORM, but is also a big issue with HTTP APIs. GraphQL and GraphQL Code Generator solve this problem beautifully, but GraphQL can still be very frustrating to work with from time to time due to its immature ecosystem. Sometimes you just want to be able to use HTTP and call it a day.

Next.JS is an amazing technology that allows us to use Typescript, but even with static typing and an ORM that has static definitions for things like a User stored in the database, that User is usually not the same User you send to the frontend as API responses so often times you end up losing that static typing along the way.

Thankfully, to solve this problem, Next.js lets us share code between our frontend and the backend where we actually don’t have to lose the type information along the way.

Here’s an example of thow that’s possible:

import { NextApiRequest, NextApiResponse } from "next"
import { db } from "../your-database-file"

const response = (id: string) => {
  return db.user.findUnique({
    where: { id },
  })
}

export default async (req: NextApiRequest, res: NextApiResponse) => {
  const { userId } = req.query

  res.json(await response(userId))
}

export type UserResponse = PromiseReturnType<typeof response>

Here we’re querying the database with Prisma which generates type information for exactly the shape of data you’re querying for the database which is why the return value isn’t explicit in the response function. It’s already inferred with the exact return value. There is a small issue however, which is that the type is actually Promise<Image> and not Image, in order to get the Image type out, we have to unwrap the Promise type. If you’re a Prisma user, then you can simply use a type-level function it exports called PromiseReturnType

import { PromiseReturnType } from "@prisma/client"

If not, you can use some manual Typescript magic instead.

export type PromiseReturnType<T> = ReturnType<T> extends Promise<infer R>
  ? R
  : never

If you’ve never looked into the infer keyword or conditional types, I highly recommend you do, they’re fantastic.

This way we can declare the types of the responses we get from an endpoint inside the endpoint file itself which makes importing much easier and also enables using type inference for database results from ORMs that are statically typed without repeating the typings.

All of this sounds pretty good so far, but when it comes to importing things, some of you might be wondering:

But not to worry, as of Typescript 3.8, we can import just types from a typescript file, meaning it’s possible to do this:

import type { HomeResponse } from "./api/home.ts"

Making sure that you don’t accidentally bring in an entire backend bundle to your frontend when trying to share types.

This is all nice and everything, but what if we could write a function that could do the work of wrapping a function that returns a promise and replying with the response value, maybe something like:

import { NextApiRequest, NextApiResponse } from "next"

type Handle<T> = (req: NextApiRequest, res: NextApiResponse) => Promise<T>

function respond<T>(f: Handle<T>) {
  return async (req: NextApiRequest, res: NextApiResponse) => {
    try {
      const result = await f(req, res)
      res.json(result)
    } catch (err) {
      res.statusCode = 500
      res.json({ message: err })
    }
  }
}

const handle: Handle = (req, res) => {
  return db.users.findMany({})
}

export type UsersResponse = PromiseReturnType<typeof handle>

export default respond(handle)

Unfortunately, this doesn’t work because of this issue about generic values. Otherwise we would be able to get the type inference working flawlessly. Typescript core team pls…

💬 Whatsapp doesn't deserve your business

Wed, 09 Dec 2020 00:00:00 GMT

Why WhatsApp Only Needs 50 Engineers for Its 900M Users

This was one of the headlines that WhatsApp was praised for in 2015, attributing it to their use of Erlang. The engineer inside me still feels giddy at the thought of this since I’m a big fan of functional programming fan myself, but something about this headline doesn’t sit right with me anymore. Every time I go to https://web.whatsapp.com to check my messages from the last 3 people in my friend circle who still use this app, and I wait for the page to load for 10 minutes only to realize that my phone hasn’t been turned on for some time —and that I need to enter my password on my phone to use a chat app on desktop— I think of this headline and it makes me angrier every time.

Laziness

The last time I was using this app regularly was around 4 years ago. That was back in the olden times when there were no stickers and ehh… uhm… yeah that’s it actually. Everything else was exactly the same. WhatsApp, with their team of the last 50 Erlang developers on Earth, have been busy doing absolutely nothing, at least on the user-facing side. Meanwhile they’ve been growing to a mind-blowing scale of 100 billion messages sent per day, which is more messages that gets sent through the global SMS infrastructure. This is even more impressive considering that they don’t allow consumer bots on their platform the way Slack or Discord does, but perhaps less so, when you realize that there is still widespread businesses automation.

The reason why this bothers me is because WhatsApp enjoys the leisure of a term I’m about to coin right now on the spot, called privacy-oriented laziness. Which is where your design decisions as a tech company based on privacy are a form of marketing for your users and a way of doing less work for your engineers. Building a system that tracks all of its users, while potentially very useful for growing a business, is a huge hassle engineering wise. Duckduckgo for example, doesn’t have to deal with the work that goes into building a personalized search engine because that’s just not what they do. WhatsApp doesn’t store any messages from users for the sake of being privacy focused but that also means they have a lot less to worry about. Of course, when you’re at the scale of WhatsApp, every technical problem is a massive challenge that requires completely rethinking the problem itself and not having to store messages doesn’t end up being as big of a win. But the domain of problems that you’re solving ends up being significantly smaller than if you had to worry about storing trillions of data points a month just for user messages alone.

It’s just messaging app bro, why feature?

Every time someone says this, the WhatsApp product manager drools all over his office desk. Why would a messaging app be subject to lesser standards you expect from other apps? I have 3 ideas for why I think this might be the case.

You haven’t experienced the features available in other, better, apps and don’t know there’s something else out there.

This is certainly possible, I didn’t have problems with WhatsApp before I discovered what Discord lets you do, now it’s indistinguishable from SMS to me.

You haven’t experienced the pain of the problems that exist in the current app.

No app is perfect, there’s always something to fix and make life easier for the users. WhatsApp is certainly in this stage and the problems it has are painfully obvious.

You are over the age of 65.

Somehow, this app found itself to be extremely popular with the older generation, which is great. I know my grandma and grandpa were baffled by Skype and thought they couldn’t do technology at all until they switched to WhatsApp and found it significantly easier and now they’re tapping away and I’m able to talk to them at a moment’s notice. Part of this coziness though, comes from the fact that WhatsApp just doesn’t change anything. Something that’s annoying enough for me to write a blog post on, is a feature in and of itself for older folks, and I of course have to recognize that.

So what are some of the things that WhatsApp is missing?

Channels
Stickers that don’t suck
Web app that doesn’t suck
Arbitrary 256 member limit for group chats
Getting your phone number (and probably your identity later) doxxed is a part of how the app works which is absolutely insane

Some of these are tied into my expectations of what I expect a chat app to do vs what some other expect it to do. It seems like for a lot of people, these things don’t matter.

Why does it matter if there are no channels if I’m just talking to my mom?

Why does it matter if there’s a group chat if I’m not in groups?

I trust everyone I use this app with so getting my phone number leaked isn’t an issue.

I understand where these arguments are coming from, you don’t have to use group chats on WhatsApp. I don’t have to use them either, but wouldn’t it be nice if there was just one app that was good that we could use for all types of chatting instead of one shitty one that sucks and another actual good one with all the features?

Alternatives

So what do you think we should use instead? Allow me to introduce you to Discord!

Actually don’t, just go look it up yourself instead, but chances are you already know about it and it’s undisputably amazing.

Maybe you might claim that

I know it's a little ridiculous to do this joke the style of a Discord chat but someone on Discord unironically argued this with me.

Other than not storing messages on a server, —which I can guarantee the people you’re talking with, including yourself don’t actually care about— Discord a superset of WhatsApp in terms of features.

Another one you could be getting into is Telegram

It too, has all the features WhatsApp has and addresses every single one of the issues above, with its cool sticker sets, 20k person group limit, hidden phone numbers and more. I feel comfortable using Telegram if Discord is indeed seen as a “not a chatting app”, which it totally isn’t but whatever.

So if all these apps have so many better features, why are people not using them? What’s the real reason people are hanging onto WhatsApp, not wanting to let go? Simply put, it’s:

The truth is, if you’ve been using WhatsApp for a long time and all your friends are on it it’s kind of difficult to coordinate everyone to jump ship at the same time. That’s the vendor lock-in effect that WhatsApp has as its biggest feature…

So while it’s true that WhatsApp revolutionized the way we do instant messaging, it just hasn’t been keeping up. Just because you’re at the top doesn’t meaan you can stop trying because someone else (cough discord, telegram cough) will be catching up with features you should’ve implemented 5 years ago.

While writing this post I sent something in WhatsApp that I had to delete later but accidentally ended up pressing

delete for me

instead of

delete for everyone

and completely lost the ability to delete it for everyone. Fuck this app AAAAAAAAAAAAAAAAAAAAAA.

🎥 A night out at the theater

Wed, 26 Aug 2020 00:00:00 GMT

If you’re a beginner at Javascript or you frequently help out beginners in online communities you know that the #1 thing that beginner programmers have problems with is Promises, or the concept of async programming in Javascript. Despite being a very useful construct in computer science, it’s not a very intuitive concept for most people learning programming for the first time.

If you’ve been linked here by me or someone else, please take the time to read this as Promise is a word you’re going to be running into for as long as you’re writing Javascript code and it will haunt you the longer you keep running away from it.

Why Do Promises Exist?

The most important thing about Promises you have to understand that promises are plans for the future, they’re not events that have been completed yet.

To better visualize what this might mean, imagine you’re running the part of the human brain responsible for making decisions. You have 3 functions you want to complete tonight called watchMovie, goHome and goToSleep. These are functions that you want to call in the order you’re thinking of. You don’t want to sleep before you watch the movie and you definitely don’t want to get behind the wheel while you’re sleeping either.

Intuitively, the way most people would approach this problem would be something straightforward like:

watchMovie()
goHome()
goToSleep()

However, these 3 are events that will take you 3 to 4 hours to complete and you, as the decision making part of the brain, have other human things to do like breathing, talking, thinking and any other activity you can think of. If you have to wait for the movie to finish before you can do any of these things again, you’re in for a lot of trouble.

In order to solve this issue you’ve decided that these functions shouldn’t completely prevent you from doing what you need to do. You should just be able to tell a part of your brain to focus on watching the movie. This way you can do things like get your nose to breathe while you’re watching the movie so you don’t die and get your mouth to tell your friend sitting next to you to stop talking so hE CAN SHUT UP WHY IS HE TALKING DURING A MOVIE THIS ISN’T A CAFETERIA JUST LET ME WATCH THE THING IN PEACE!

This also means that when you call these special functions, you’re actually just sending them off to different parts of your brain to be done later and moving on ASAP since you have other things to be scheduling. Because of that, running the earlier code:

watchMovie()
goHome()
goToSleep()

Now means that you’re getting a part of your brain to watch a movie, get in your car to drive home and also sleep at the same time. Not what we were trying to do at all.

Sequencing Different Actions Together

When you’re asking different parts of your brain to do these, you get a receipt or quite literally, a Promise, back saying that the thing you wanted to happen will be done. You will also be notified when it’s completed so you can do other actions afterwards.

This is exactly the function of .then() in Javascript. It lets you run a function only after a “future” action like watching a movie has finished. With that in mind, we can now chain watching a movie and going home like

If you're not familiar with

() => {}

I recommend looking into arrow functions and more modern ES6 syntax.

watchMovie().then(() => {
  goHome()
})

What if watchMovie() returned some ideas about the movie and you wanted to go to sleep, dreaming about those? It might be tempting to do something like:

let dreams = null
watchMovie().then(movieThoughts => {
  dreams = movieThoughts
  goHome()
})
goToSleep(dreams)

But .then() only runs when the movie has finished running and like we established above, you have to move on with the things you do so in this scenario you’re going to tell a part of your brain to watch a movie and then immediately tell another part of your brain to go to sleep with null dreams since you didn’t wait for the movie to finish

Instead the correct way to do it would be

watchMovie().then(movieThoughts => {
  goHome().then(() => {
    goToSleep(movieThoughts)
  })
})

With this you’re waiting for the previous scheduled promise to finish before running the next ones.

This isn’t going to be a full tutorial on Promises, if you’re looking for more resources you can check out these resources.

https://ui.dev/async-javascript-from-callbacks-to-promises-to-async-await/

https://javascript.info/promise-basics

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise

And my personal favorite:

Reason for Promises in Real Javascript

The reason why Javascript behaves this way by default and why a language like Python or Java doesn’t is because even though Node is a big player in server side technology these days, Javascript was historically meant to only be run in a browser. And in a browser, Javascript is tasked with re-rendering the HTML on your screen and scheduling other things about web pages. If every time you make a request, Javascript has to wait 1+ second for the request to finish, much like how your brain wouldn’t be able to breathe, Javascript won’t be able to paint changes to the screen like animations, button clicks, typing and so on, leading to terrible user experience.

Promises are a confusing concept so if you’re still confused, don’t worry too much about it. This idea is going to click as you keep working with it more and more.

🤡 Every technology deserves to be memed

Fri, 17 Jul 2020 00:00:00 GMT

Ben Awad, a tech Youtuber I’ve been following for a while who mocks

A Javascript framework made by Google. It's not as popular as others, but still widely used around the world.

and its users as a running joke on his Youtube channel and Twitter recently released a video explaining how what he’s been doing might be harmful after receiving criticism from the

A Javascript framework like Angular, but is more popular and less opinionated.

god himself.

In the video and the discussion on Twitter, the 2 ideas of making low effort jokes that get tiring over time and making harmful jokes seem to be conflated and I’ll try to address that later on.

I believe the attitude that jokes about Angular or any other technologies is harming that community is problematic for different reasons but before you agree with me, I want to preface this by saying this isn’t a political take. I’m not “tired of snowflakes” and if you agree with me because you want everyone to stop being too sensitive, you should probably stop reading now.

Muh technology

Over the past 2 years I’ve had the privilege of moderating a programming community on

A chat platform like WhatsApp or Skype that was originally aimed towards gamers but has since grown way past gaming.

that’s now grown to over 50k members as well as being a member of a couple other ones. Undoubtedly, I learned a lot about programming during the years but I also learned a lot about how people view and talk about it.

There’s unfortunately a large chunk of programmers in these communities ranging from complete beginners to seasoned veterans who view their favorite languages and frameworks not as tools designed for specific tasks, but as lifelong companions. Watching programmers argue about whether Python is a “good beginner’s language” feels more akin to listening to your dad and uncle shout at each other across the living room during a soccer tournament rather than a reasonable discussion between experts (which is why we no longer allow discussions about what language beginners should start with anymore).

We spend a lot of time around our favorite tools, setting up

A collection of configurations saved online to allow devs to reinstall their preferred tools quickly on new machines.

to make sure we’re not separated from our preferred technologies and working on getting better at a single programming language instead of hopping between many different ones to learn more. While none of these are unreasonable, it gets us more attached to our preferences and it leads to conflict in tech communities combined with other factors like:

Developers often become confident in their opinions, choices and preferences as they improve, sometimes becoming too confident.
Feeling like you’re a part of a community of unique people and seeing yourself as an Angular developer feels nicer than being just any other developer who uses whatever language is fit for the job.
People are just very impressionable in general.

That last one resonates quite a bit with me. When taking my second programming class in a community college, I would often hear people who I knew were beginners say things like “Yeah man Python’s actually a kid’s language, C++ is much harder and just better” while the people around him who’ve also never used any language other than the one we were learning in class nodded in agreement. This was after I’ve already been exposed to the same kinds of arguments online so it wasn’t much of a shock to me but still frustrated me.

Don’t fall in love with your tools

So what does any of this have to do with making fun of technologies? I think tool attachment and advocacy is a serious problem in tech communities that prevents us from improving as developers and very often leads to people being defensive about tools they like and aggressive towards ones they don’t. There’s a very interesting article by Mark-Jason Dominus that approaches the same problem through the perspective of Perl, a language that’s fallen out of favor significantly among most developers by now, 2 decades after the article was written, but still had the zealots making the same arguments at the time.

Because of this, when I see replies like

Honestly man, this is not cool. It’s been kind of depressing to see you bash anything that starts with an A. I’m not the one to tell others about low effort posting but this is just tired.
— Dan Abramov (@dan_abramov) July 18, 2020

I say let the kids play Dan, let’s make fun of Angular and React and anything else. Let’s show that as developers, we don’t to create an identity based around tools. Technologies come and go and we don’t need to form emotional attachments to them, a joke or criticism aimed towards the tool of your choice isn’t an insult towards who you are. You can be a proud Vue developer or a carrot farmer, it just doesn’t matter because at the end of the day, you’re a developer like me. Sure it feels good to view yourself as a member of a community of developers who use the same tools as you, but you can do that without feeling like you have to pledge an allegiance to it.

I believe that becoming more comfortable with jokes thrown around at different technologies will prevent new, impressionable developers from blindly taking sides with the ones they personally enjoy using if they see that devs make fun of the idea of a single, superior tool.

Online circlejerks

I will admit, however, that the one problem with running amok with jokes is that when making them, the fact that they’re jokes has to be communicated clearly. It’s possible that jokes like these might still devolve into

When multiple people on the internet who have a shared interest or ideology get together to congratulate each other for having the opinions they do.

_{Please just trust me on the definition and don't Google this word}

where people repeat the same old incorrect things that are no longer meant to be funny and end up having the opposite intended effect.

A rare image of a highly sophisticated discussion about languages in an online programming community.

I definitely don’t want to encourage this, but I think that if people see that other programmers don’t take themselves and their tools too seriously, this will not happen.

I’m serious. I’ve seen a variation of tweets that are all like “angular bad” over the last few days and I think this lazy attempt at humor makes our community look like jerks and creates artificial division.
— Dan Abramov (@dan_abramov) July 18, 2020

I can understand where Dan is coming from with this. Like I said before, jokes can get out of hand and turn into something harmful and it’s definitely a good idea to consider that. I just don’t know if artificial division is a valid concern, it’s not like joking about programming tools is the same as joking about one’s gender or race. It’s not what makes you you and there’s definitely merit in separating your feelings from your tools.

Coming back to the idea of conflating bad jokes with harmful ones. I think, yes, Ben’s Angular jokes probably got a little bit too out of hand and it ended up being more annoying than funny for some (though I personally still enjoy them sorry not sorry). Dan’s comment of low effort posting is definitely valid criticism. I just don’t think it was necessarily harmful towards any particular group that needs protection.

So keep poking fun at different technologies and being funny. When you’re at the receiving end of these, you need to remember that at the end of the day you are who you are and not what you use. Consider everything with a critical mindset and don’t take yourself too seriously.

💀 The terrible fate of reduce

Tue, 14 Jul 2020 00:00:00 GMT

I recently came across this video on youtube that a friend linked talking about the problems with reduce:

and it gave me some conflicting thoughts which got me thinking. Why is it that the tool that plays such a crucial part in functional programming so hated by experienced developers? Of course, debating menial stuff is a programmer’s favorite pastime but the reduce argument seems to be popping up every single time someone suggests using reduce to solve a problem.

Before continuing, lets briefly go over what reduce actually is

function sumNumbers(numbers: number[]) {
  return numbers.reduce((sum, num) => sum + num, 0)
}

Here, reduce simply takes a list of numbers and goes over them, while changing the accumulator for the next iteration. This is, of course, an oversimplified example but the basic idea is that you have a generic list of items, and there’s an accumulator of some data type that is updated in each iteration. In that case, what’s so bad about it?

Hate the player, not the game

From the video and the discussions I’ve had with people online, it seems like a lot of the hate that’s directed towards reduce is that it’s easy to abuse. There are a lot of ways to use reduce in ways it wasn’t meant to be used, though you could say that it’s possible to blame a lot of different things with that reasoning. It’s not the function itself that’s problematic, it’s just ways that people use it. On one hand that’s true and doesn’t make reduce itself bad, but on the other, programmers read other programmers code and the way people use tools affects the way it’s viewed as well.

But the story of reduce starts way before Javascript, so lets take a look at the origins of reduce itself and why it feels like a monstrosity to developers.

Functional Origins

Reduce is actually somewhat of a more modern name for the same concept found in many functional languages referred to as a fold. Like many other popular features like lambdas and higher order functions, the idea of folding is borrowed from languages like Haskell and Lisp because they are very convenient to use in those languages.

However, I want to argue that the convenience of functions and concepts in paradigms aren’t solely based on the merit of the idea itself, but are closely tied to how they fit within the paradigm they’re used in.

Consider the Array.prototype.flat rewritten in Typescript that’s written using reduce

function flat<T>(arrays: T[][]): T[] {
  return arrays.reduce((all, arr) => all.concat(arr), [])
}

It’s not too bad, I would argue reduce is the right tool for a function like this, and now the same function written with a regular for loop

function flat<T>(arrays: T[][]): T[] {
  const out: T[] = []
  for (const arr of arrays) {
    out.push(...arr)
  }

  return out
}

You could argue that the for loop version is longer-ish or that the reduce doesn’t give you any benefit and both of those would be reasonable thoughts. If reduce is such a “good” concept, why is this a controversial argument? Higher order functions are a good concept and we don’t spend nearly as much time debating those.

The answer to this is in how JS and other popular languages try to fit features that don’t work well in non-functional paradigms in their language.

Reduce sucks but fold is king

flat :: [[a]] -> [a]
flat = foldl (++) []

This is the Haskell version of the same function above and it looks… comical in comparison. Some questions that may arise in reader’s mind when looking at this example are:

How is this so much shorter than the JS version?
What the hell is going on here?

We’re essentially doing the exact same thing as the first one JS example

flat :: [[a]] -> [a]

We’re stating that a flat function takes an input of 2 nested arrays and returns a singly nested array:

flat = foldl (++) []

And that flat is defined as a reduce of the input where the function is (++) (array concatenation) and the initial value is [], an empty array.

The reason why this is so much shorter than the JS counterpart doesn’t actually have anything to do with Haskell specific syntax. It’s possible to write reduce in this exact way in Javascript

const flat = arr => reduce(concat)([])(arr)
// or we can reduce this definition to just
const flat = reduce(concat)([])

flat([
  [1, 2, 3],
  [4, 5],
])
// [1, 2, 3, 4, 5]

But we choose not to because it doesn’t feel natural in JS (Also Typescript, as of today with 3.9 isn’t able to infer types of curried functions very well but that’s a different story). If you’re an epic category theorist this is fine but it can be a problem if your coworker submits a pull request with code above and you can’t help but feel like he’s challenging you to a high IQ competition. It seems like a big part of the debate revolves around people who are used to a functional style trying to shoehorn functional patterns into every language they use vs js programmers who are either not familiar with FP or don’t want to use it in JS.

So what do we do

It seems like the syntax alone does not solve this problem. Fold/Reduce seems to be a good concept that works in the right languages, but not so much in javascript.

Warning

Opinion Ahead

I think reduce is fine to use, don’t try to use it like you’re writing Haskell but the idea is valuable. If your code is too smart, that’s a problem that goes beyond just reduce. Reduce is innocent.

I also think Javascript should have put the seed value of reduce as the first argument, not the second. In functional languages with currying, it’s valuable to have the seed value as the second argument because your code is more modular when you leave the data to be inputted last so you can pass it as an argument later, but this has no use with JS. It would’ve been much better to have reduce that goes:

function sum(nums: number[]): number {
  return number.reduce(0, (all, num) => {
    return all + num
  })
}

This way the initial value and the function are all in the same place and you don’t have to go up and down the function call to see what you’re working with.

In essence, just be responsible with your code and it doesn’t matter whether you’re using reduce or the fancy function from next hot new paradigm.

🧠 Learning for fun and profit

Thu, 09 Jul 2020 00:00:00 GMT

You’re a 19 year old highschool graduate, unsafisfied with life and with no desire to do anything, partially because you don’t know what’s out there, but also because you see that you get bored of everything you’ve tried to learn in the past. You feel like you’re a curious person but not a hardworker because you’ve never seen yourself succeed in things other people work hard to get better at, especially not school. The one thing you do love is gaming but you’re not even successful at that despite pouring hundreds, no.. thousands of hours into it. Knowing this about yourself, you think “maybe I’m just bad at learning”, and proceed to log back into World of Warcraft for the millionth time to forget about your problems.

Future in shambles but that loadout popping off though

6 months after starting a chemistry degree you realize you hate it and drop out, losing any little certainty you had about the future of your life. Chemistry was your strongest subject so now you’re left with absolutely nothing else to do. You eventually decide that you’re going to sign up for a computer science degree and you come to this brilliant conclusion by remembering that you’ve tried programming 3 times before and hated it every time but maybe it’ll be ok because you like playing games on the computer and programming is also done on the computer…? Not the smartest choice you’ve made, but a choice nonetheless.

Fast forward a week or so, and you’ve decided that you might as well get started and look up some programming stuff and you remember hearing that Python is a good beginners language, so you figure you might as well get started with it. You pick up Zed Shaw’s awful Learn Python the Hard Way and start reading through a fairly bland book.

print("Hello World")

Cool, you wrote your first line of code but it wasn’t all that impressive and you get bored about an hour in, but this is a kind of boredom you haven’t felt before. You don’t feel an urge to close the tab and start a new game of League, you want to keep writing code. It is here that you do something unspeakable, something so outrageous that it makes all highschool teachers and college professors around the world turn in their metaphorical graves. You decide to skip ahead and do something more interesting instead of continuing to learn the basics. This isn’t how you’re supposed to do things, maybe you’re just bad at learning.

A day or two go by and after making a coinflip and number guessing game you’re bored once again even though working on them was enjoyable. You’ve recently been spending a lot of time on Discord and realized that the music bot you’ve been running with your friends is written in Python! You go through the source code to see how it works and by some miracle you somehow figure out how to add a new command. It’s not pretty, and definitely cursed but you managed to use your 3 day programming knowledge to make something happen. Your friends don’t seem to care too much but it doesn’t even matter, you’re satisfied after you managed to get a language you barely know working with an API you’ve never used before. Wanting to do even more exciting things, you continue to write code but constantly hit road blocks, you eventually decide to ask for help but the people end up telling you you suck and need to learn how to program… maybe you’re just bad at learning.

But you’ve found something you find fun! So you keep going and eventually writing code becomes so enjoyable that you no longer have urges to play games all day, or even at all. Your desire to keep writing discord bots forces you to look up a bunch of things you didn’t know, you end up learning about promises, how to use a database, you finally understand how websites work and even attempt making one yourself to add a game in it.

But it sucks and you’re not happy with it and you know nobody else would be either. As if making games isn’t hard enough, you’re frustrated that your hard-earned programming knowledge isn’t helping you make your websites look half-decent. You keep up the hard work but every new design is a disaster more terrifying than the last and you’re not making any progress, maybe you’re just bad at learning.

A year passes and despite having a bunch of projects under your belt you look back and see that almost all of them are unfinished and abandoned projects you either gave up on or got bored of but at least you’re getting the hang of things now and continue working on things that excite you like you’ve been doing since day 1. Eventually you get around to learning things like React, Angular, Vue, MongoDB, your curiousity for programming spreads beyond your initial goal of web development and you find yourself looking up the documentation for different popular languages like C++ or Java but also more obscure stuff like Haskell, Purescript or Elm just because the way it works blew your mind and no other reason.

spongebobCase :: String -> String
spongebobCase = zipWith ($) (cycle [toLower, toUpper])
-- spongebobCase "memes" => "mEmEs"

Math, a thing you once swore to never touch unless your life depended on it, suddenly became more interesting because of something you didn’t even know qualifies as math. You even started to earn money doing something you love and all this time you’ve been begrudgingly attending school, skipping classes (even computer science classes) to continue working on your projects. You finally take a step back and look at the progress you’ve made, you feel as if you’re capable of doing so much more than you ever thought you could be, and you think about the time you felt like an idiot in highschool for not being able to keep up in classes. How you weren’t able to get anywhere with things your parents and teachers kept telling you were important and you realize maybe you’re not bad at learning, maybe they’re bad at teaching…

But it only takes one to tango.

🔎 Asking good questions

Thu, 07 May 2020 00:00:00 GMT

Regardless of whether you’re a complete beginner or an experienced programmer, chances are you’ve had to ask questions before. The most common form of this is going on Google to look up an error message, something you’ve surely done by now. You’ve probably also noticed that there are some ways you can formulate your questions on search engines that will give you better results than others like searching for “add element list python” as opposed to “how do I add an element to a list in python” (although search engines have gotten really good at handling the latter one nowadays too). But in a lot of cases, just Googling the error doesn’t help us so we resort to asking a real person, and much like a search engine, there are ways to make your experience with this much more pleasant by changing the way you form your question.

What’s a bad question

Before we can understand how to start asking good questions, we have to go over what a bad one looks like so we can avoid it in the future. Bad questions are considered bad because they do not communicate the intended idea efficiently or clearly.

Truly bad questions lead to a lot of back and forth between the person who needs help and the helper, which often start to resemble a police interrogation where the asker tries to withhold as much information as possible (for some reason) and must be asked for details one by one instead of providing it themselves.

Examples of bad questions

AmazingProgrammer: Any javascript pros in chat?

FantasticProgrammer: I’m making a bot and it gives some errors, anyone help?

IncredibleProgrammer: I’m trying to send a message but it says cannot read property 'send' of undefined, help?

PhenomenalProgrammer: This isn’t working.
async def brrr():
  pass

brrr(money_printer)

GreatProgrammer How to install node.js in ubuntu?

Let’s go over what’s wrong with each one of these.

Asking to ask

Amazing programmer has a question but they haven’t even started asking it. It’s the question before the question. This wastes your own time as well as the helpers time, don’t ask to ask. If you just post your question in one go, by the time someone who can help you sees it their response won’t be “yes?”, it will probably be the solution to your problem.

No detail

This usually comes from frustration but FantasticProgrammer hasn’t provided even the slightest bit of information about what the problem is. It’s impossible to help them without a followup question.

Insufficient detail

IncredibleProgrammer provides more information than the previous person and sometimes this will be such a common problem that someone will be able to tell them what the problem is with almost no context but that’s rarely the case and figuring out what the problem is from just this much information is difficult.

Missing context

A code sample helps, but helpers who haven’t mastered mind reading need more context to be able to offer a solution to your problem. “It’s not working” isn’t enough context, make sure you’re specific about your problems and what your expected outcomes are unlike PhenomenalProgrammer.

No effort

Keep in mind that people who are helping you are people and not a search engine. “How to install node.js in ubuntu?” is a question that should be asked on Google and not towards real people. If you have problems with installing node.js after Googling, you should definitely include that in your question so it’s easier to know where you are at with the problem without needing to prompt you for more details. Help the people who want to help you.

What’s a good question

Here’s a mental checklist that I try to keep track of when asking for help.

Written in one single message
Has sources for things I’ve looked up
Error messages (if any)
What I expected to happen
What actually happened