Ayase Imageboard Archival Standard (Ayase)


Reference Implementation

Note that no Ayase is still in the drafting stages, and some candidates have been identified but nothing has been determined for use yet.

  • Database: PostgreSQL - Probably not with JSONB thinking about it honestly, 4chan is a relational db forum in the first place.
  • Scraper:
    1. Ena (Rust) - A current top candidate for the Ayase schema.
    2. unnamed go scraper - secrets abound here
    3. Hayden (.NET C#) - Asagi drop in replacement is earmarked as a high priority, so Ayase support is currently on the backburner, but it could be retooled for this purpose or even as a base for a future conversion methodology.
    4. Eve (Python) - A quick and dirty python scraper compatible with Asagi, but not fully finished and kind of stalled.
  • Middleware: Ayase (Python PyPy)
  • Frontends: 4chan X (4plebs-x), Clover, iphone app


Full Images


Note that it is only possible to get a sha256sum of full images when doing full image scraping. Thumbs only scrapers will have to follow the procedure noted in the next section.

    • This is to mitigate the ongoing issue of md5sum collisions becoming ever easier for users to do. However, this does require migration of filenames to sha256sum and their calculation, but on powerful server hardware especially with hardware acceleration, it should not be feared. More on md5 collisions as they relate to 4Chan and its archives: Ayase/MD5_Collisions
    • The trustworthiness of the sha256sum is strong up to the dawn of quantum computing it seems. By that point if it is cracked there will be more issues than mere collisions to worry about.

Thumbs Only Scraping

When thumbs only scraping is used, it is not possible to record the sha256sum of the target image, so the sha256 key in posts is left undefined.

Nevertheless, sha256sum for thumbnails should always be generated and stored under the key sha256t. This checksum refers to the thumbnails own sha256sum filename and is unrelated to the image filename.

Third party 4chan clients wishing to add support for the 4chan archives would therefore use sha256t + .jpg as the thumb filename instead of tim + s.jpg. To provide backwards compatibility though, the foolfuuka reference style desuarchive.org/thumb/1234/56/123456789s.jpg can be used to resolve to whatever the file is currently named, whether under the asagi way or the sha256sum way. This is to allow seamless support while images get renamed, and ensure that urls never break.



One issue with converting to PostgreSQL timezones is that the legal definition of time for a city is not as simple as a single component timezone such as EST, as it sometimes becomes EDT for months and then switches back. Instead, the GNU tz time locale America_NewYork is used.

Another problem is that the conversion asagi did is lossy, it does not store whether the unix timestamp refers to EST or EDT. This means that special scripting to determine the proper timezone based on whether the calendar date is legally a time change day: whenever the next post would have fallen back an hour, that indicates a daylight savings time change. Next, inference from the post number order context (less accurate but only small edge cases) will be necessary for time changes forward to ensure that the hour forward is not due to 4chan being down for a period.

Therefore any conversions will need to carefully consider this by checking the true legal timezone for the tz locale America_NewYork on the calendar date when storing a timezone. 



  • In Asagi: posts are not made up of the raw data from the 4chan API, which includes HTML escapes. They are instead stored unescaped after processing by the function
    This may have been due to its historical use as an HTML scraper. Whether we should continue to replicate this is an open question as it does not seem to have major data loss and should still be compatible as output, but my opinion is that especially as it will be put into json, we should escape it properly so that the JSON engine does not have to do it by itself.

Since cleaning html by regex is a lossy conversion, best to leave asagi posts the way they are cleaned, but all future posts just leave it verbatim from the 4chan api, with only simple security html tag removals if at all necessary (4chan does some already).

PostgreSQL JSONB Schema




PostgreSQL RBAC Row Permission System





Official Documentation:




Elasticsearch Engine

Elasticsearch is a NoSQL DB with a focus on search functionality. The existing archiver stack search system, based on Sphinxsearch (running on the archive's MySQL DB), has frequently been a limiting factor for various 4chan archives (slowdowns, search disabled on certain boards etc.) It is believed that moving the search system to Elasticsearch could alleviate some performance issues. It would also theoretically allow for horizontal scaling of search with increasing post volume, and to split archive functions across globally-distributed hardware resources, among other benefits.

An ongoing effort towards a "greenfield"-style Elasticsearch-based archiving project is hosted in these repos for a scraper and search stuff.


  • Add an Elasticsearch search layer to the existing stack, replacing Sphinxsearch, but only use it to return matching post IDs
    • May be the easiest option
    • Would be business as usual for non-search DB operations
    • Would require full in-memory duplication of contents in SQL DB and Elasticsearch (ES synced with MySQL)
    • Retain original post data retention/backup scheme, durability
  • Add an Elasticsearch search layer to the existing stack, replacing Sphinxsearch, and use it to return entire documents/posts
    • Might be more intrusive than the above
    • Would still be business as usual for non-search DB operations
    • Would still require full duplication of contents in SQL DB and Elasticsearch on-disk, but post contents may not need to be held in-memory on the SQL side
    • Retain original post data retention/backup scheme, durability
  • Replace the existing SQL DB completely with Elasticsearch
    • Possible long-term option, not the best to serve immediate needs
    • Would require rewrites of any DB-facing component, not just search
    • Needs new data retention/backup scheme


Determine which of the two "Elasticsearch to complement existing SQL DB" options is more practical in the short term, develop a proof of concept. Run modded Foolfuuka, unmodded Asagi on a couple low-traffic boards with the new Elasticsearch layer. [Link to Kanban board etc.]