The collapse and rebirth of a six-year-old open source project

I have an open source project that I have maintained for six years, RSSHub, and it is facing collapse

Background

On the surface, it has nearly 30k Stars, more than 900 Contributors, more than 300 million requests per month and countless users, dozens of dollars of monthly sponsorships, a steady stream of issues and PRs, and the code is updated almost every day. It is very healthy and dynamic, but invisibly, the high maintenance time cost that lasts for several years, the server cost of more than a thousand dollars per month, and the tedious and gradually accumulating maintenance work repeated every day, make it repeatedly jump on the edge of collapse.

The project was developed six years ago. Many of the fashionable Node.js technology stacks and dependency libraries with the slogan “Next Generation” at that time have become tears of the times. They are very old now and many popular new technologies cannot be applied, such as JSX, TypeScript, Serverless, etc.; its architecture is also very unreasonable. The information of each route is scattered in multiple places. Developing or changing a route requires multiple modifications. There is one place to register the route, one place to write the routing script, and one place to write Radar. Rules, one place to write documentation… This adds a lot of work and is prone to errors. It was not a problem when there were few routes, but now it has become unbearable

Maintaining the status quo under such a poor infrastructure is already a struggle. Developing new features is even more difficult. It will only increase the difficulty of future updates, so it is difficult to realize the novel ideas that sometimes pop up in my mind.

The only way to solve these problems is to rewrite the kernel using a modern framework and a newly designed architecture. However, as there are more and more routes, the cost of transformation is getting higher and higher. Each basic change may require up to several months of work. Therefore, although the problem is becoming more and more serious, we continue to postpone it based on the principle that it is not unusable.

But it had to be done, so I took the time to spend a few months redesigning and rewriting it

Technology stack update

koa -> Hono

The first step, which is also the most basic and difficult, is to replace the previously used Web framework koa. As a next-generation web framework that was popular six years ago, the author has long since abandoned it. After research, he decided to switch to Hono, which has the best support for JSX, TypeScript, and Serverless.

Their APIs are very different and require rewriting all middleware and replacing the koa API used in all routes

Major changes: https://github.com/DIYgod/RSSHub/pull/14295

Hono author also likes this transformation

https://twitter.com/yusukebe/status/1762801106340782222

JavaScript -> TypeScript

Switching to TypeScript can avoid many type problems and low-level errors. The most important thing is to ensure that hundreds of contributors are consistent and difficult to make mistakes, and the quality of routing code for subsequent contributions is not too bad.

Major changes:

https://twitter.com/DIYgod/status/1764360942035312879

CommonJS -> ESM

ESM is a specification strongly promoted by some Node.js core developers a few years ago. It has some advantages, but most of them are criticized for ecological fragmentation and functional simplification caused by incompatibility with the previous CommonJS.

After several years of development, it can now be said that most scenarios are barely usable. tsx also provides support for scenarios where CommonJS and ESM are mixed.

Although we have tried our best, there are still some CommonJS codes that are temporarily difficult to migrate. As a result, they can only be run using tsx and are incompatible with some Serverless such as Vercel. However, there is still an opportunity to solve it slowly in the future.

Major changes:

art-template -> JSX

art-template is a template engine that supports koa. I remember there was a more popular template engine six years ago, but I can’t remember the name. I chose art-template because I didn’t understand the more popular one at the time. It was very simple.

Hono comes with JSX support. There is no need to introduce JSX. The syntax extension of JavaScript is equivalent to using React.

Major changes:

Jest -> Vitest

Jest was once a popular testing framework, but it has become less and less popular after the advent of the ESM era. The support for ESM has always been “experimental support”, and now Vitest is more popular.

Major changes: https://github.com/DIYgod/RSSHub/commit/38e42156a0622a2cd09f328d2d60623813b8df28

Got -> ?

The currently used Got is no longer actively maintained, and no good replacement has been found. It may be replaced with native Fetch or self-encapsulated Fetch in the future, but I haven’t started yet.

New routing standard

My own abilities are still not enough. I have learned and improved a lot during discussions with community developers. The process is very interesting: https://github.com/DIYgod/RSSHub/issues/14685

Major changes: https://github.com/DIYgod/RSSHub/pull/14718

History

The new standard is mainly to solve the problem of too scattered routing information. This should be considered the third edition.

The first version came from the development stage of RSSHub. At that time, it was not foreseen that there would be so many routes, so there was almost no planning. All routes were registered in the same file, and then routing scripts and documents were added. Later, the file became larger and larger, and it was easy to conflict. In addition, all routing scripts were loaded during the startup phase, and the program performance became worse and worse.

The second version comes from the maintenance period of NeverBehave. Namespaces were introduced and router.js and radar.js were cut. Routes in the same namespace were concentrated in the same folder and one or more Markdown documents. Lazy loading was also implemented, which greatly improved maintainability and performance. However, it was still scattered in multiple files, and information in different files was also prone to inconsistencies leading to errors.

Now

This time, routing files are divided into two categories, namespace.ts and routing files with any name.

namespace.ts will define the namespace information by exporting an object named namespace

import type { Namespace } from "@/types";

export const namespace: Namespace = {
  // ...
};

The fields contained in the namespace are restricted via TypeScript to

interface Namespace {
  name: string;
  url?: string;
  categories?: string[];
  description?: string;
}

This information will be compiled and used by documentation and RSSHub Radar

The routing file defines routing information by exporting an object named route.

import { Route } from "@/types";

export const route: Route = {
  // ...
};

The fields contained in route are restricted via TypeScript to

interface Route {
  path: string | string[];
  name: string;
  url?: string;
  maintainers: string[];
  handler: (ctx: Context) => Promise<Data> | Data;
  example: string;
  parameters?: Record<string, string>;
  description?: string;
  categories?: string[];

  features: {
    requireConfig?: string[] | false;
    requirePuppeteer?: boolean;
    antiCrawler?: boolean;
    supportRadar?: boolean;
    supportBT?: boolean;
    supportPodcast?: boolean;
    supportScihub?: boolean;
  };
  radar?: {
    source: string[];
    target?: string;
  };
}

Previously, route.js, mantainer.js, radar.js and document information were concentrated in this file, reducing multiple definitions and reducing the possibility of errors.

Implementation

The implementation logic is that the development environment traverses the entire route folder, finds all namespace.ts and routing files, reads information, loads routes, and uses a pre-compiled path list in the generation environment to avoid traversal and unnecessary loading processes. The code is at: https://github.com/DIYgod/RSSHub/blob/master/lib/registry.ts

The document is also created by traversing the route folder, finding all the required information and then synthesizing it into a series of Markdown files. Manual maintenance is no longer required. The code is at: https://github.com/DIYgod/RSSHub/blob/master/scripts/workflow/build-routes.ts

Of course, routes developed using previous routing standards need to be migrated to the new standard rather than abandoned directly. The information has been captured and organized in batches through scripts and replaced. However, especially the documents are confusing and contain many errors, so the captured information also has many errors, which can only be manually modified gradually in the future.

future

Through this series of improvements, RSSHub can finally throw away its historical baggage and develop new features with peace of mind. Here are some of the ideas I have accumulated to inspire others:

Since RSSHub is a data collection, its use is not necessarily limited to RSS. The JSON output function can be enhanced and used as a general RESTful API. For example, it can provide an interface to get the next page or output non-feed data like Twitter follower count.
User system and user-defined configuration, generate your own private subscription address #14706
Routing error notification and health detection #14712
Linkage with RSS3 nodes and cryptocurrency revenue sharing https://twitter.com/rss3_/status/1731822029199094012
AI translation and summarization
More detailed analysis of instance data and reverse derivation of automatically recommended Radar rules
An RSSHub instance bound to a local browser or client is expected to truly solve the problem of anti-crawling -…

Finally, open source is a very expensive thing, and RSSHub cannot survive today without the help of these developers.

and these kind-hearted people who sponsored

If RSSHub is helping you, I hope you can actively participate and contribute your own small force to the future of information freedom.