Skip to content
Back to search

backend-next

The refactored Penguin Statistics v3 Backend. Built with Go, fiber, bun and go.uber.org/fx. Uses NATS as MQ and Redis as state synchronization.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/penguin-statistics/backend-next with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/penguin-statistics/backend-next'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/penguin-statistics--backend-next/badge.svg)](/listings/penguin-statistics--backend-next)

HTML

<a href="/listings/penguin-statistics--backend-next"><img src="/api/listings/penguin-statistics--backend-next/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 145 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <img src="https://penguin.upyun.galvincdn.com/logos/penguin_stats_logo.png"
  2. alt="Penguin Statistics - Logo"
  3. width="96px" />
  4. # Penguin Statistics - V3 Backend
  5. [![License](https://img.shields.io/github/license/penguin-statistics/backend-next)](https://github.com/penguin-statistics/backend-next/blob/main/LICENSE)
  6. [![Last Commit](https://img.shields.io/github/last-commit/penguin-statistics/backend-next)](https://github.com/penguin-statistics/backend-next/commits/dev)
  7. [![GitHub Actions Status](https://github.com/penguin-statistics/backend-next/actions/workflows/build-release.yml/badge.svg)](https://github.com/penguin-statistics/backend-next/actions/workflows/build-release.yml)
  8. [![go.mod Version](https://img.shields.io/github/go-mod/go-version/penguin-statistics/backend-next)](https://github.com/penguin-statistics/backend-next/blob/main/go.mod)
  9. [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fpenguin-statistics%2Fbackend-next.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fpenguin-statistics%2Fbackend-next?ref=badge_shield)
  10. ![Alt](https://repobeats.axiom.co/api/embed/ae259f962b4d5fc542de420d31a237986d282765.svg "Repobeats analytics image")
  11. The re-designed [Penguin Statistics](https://penguin-stats.io/?utm_source=github) v3 Backend, built with Go.
  12. ## Using the data
  13. The drop data is public:
  14. - **REST API** at `https://penguin-stats.io/PenguinStats/api/v2/`: see the
  15. [developer docs](https://developer.penguin-stats.io/public-api) and the
  16. [OpenAPI (Swagger 2.0) spec](https://penguin-stats.io/swagger/doc.json).
  17. - **MCP server** for AI assistants (Claude, ChatGPT, Cursor, VS Code, Cherry Studio and others) at
  18. `https://mcp.penguin-stats.io/mcp`: Streamable HTTP, no authentication, read-only. In Claude Code:
  19. ```sh
  20. claude mcp add --transport http --scope user penguin-stats https://mcp.penguin-stats.io/mcp
  21. ```
  22. The steps for every other app, in English and Chinese, are at https://mcp.penguin-stats.io/.
  23. - **Plugin and skill** for coding agents, the MCP server with a skill on reading, citing and querying the numbers:
  24. https://github.com/penguin-statistics/agent-skills
  25. The drop data is licensed [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/): non-commercial use, with
  26. credit. Credit it as `Data: Penguin Statistics (penguin-stats.io), CC BY-NC 4.0`. The code of this repository is
  27. [MIT](LICENSE).
  28. ## Architecture
  29. This Go project mainly uses:
  30. - [gofiber/fiber](https://github.com/gofiber/fiber)
  31. - [uber-go/fx](https://github.com/uber-go/fx)
  32. - [uptrace/bun](https://github.com/uptrace/bun)
  33. This project mainly follows a RESTful API design.
  34. ## Operations
  35. ### Report tasks in the dead-letter stream
  36. The report worker moves a report task out of the `penguin-reports` stream into `penguin-reports-dead` (subjects
  37. `REPORT_DEAD.*`) when it can never be processed (it is not a task, or a stage of it does not exist) or when it failed at
  38. each of its 30 deliveries, which take about 97 minutes of retry delays. Each move is logged at error level
  39. (`reportwkr.malformed`, `reportwkr.parked` or `reportwkr.dead`). The copy keeps the payload as published, with these
  40. headers:
  41. - `Penguin-Original-Subject`: the subject it was published under, like `REPORT.SINGLE`
  42. - `Penguin-Original-Sequence`: its sequence in `penguin-reports`
  43. - `Penguin-Deliveries`: its deliveries when it moved
  44. - `Penguin-Last-Error`: the error that moved it
  45. A task too close to the maximum payload to fit with these headers moves without them, and its subject under
  46. `REPORT_DEAD.*` names its original subject; its move is logged as `reportwkr.dead.bare`, with the sequence and the error.
  47. Nothing in the stream expires. With the [nats CLI](https://github.com/nats-io/natscli), once the cause is fixed:
  48. ```sh
  49. # list the tasks, with their headers
  50. nats stream view penguin-reports-dead
  51. # show one, by its sequence in penguin-reports-dead
  52. nats stream get penguin-reports-dead 42
  53. # replay it under its original subject: the worker processes it as a new task
  54. nats stream get penguin-reports-dead 42 --json | jq -r .data | base64 -d \
  55. | nats pub REPORT.SINGLE --force-stdin
  56. # remove it once replayed
  57. nats stream rmm penguin-reports-dead 42 -f
  58. ```
  59. A task that moved after failing at each delivery may have been stored by one of them whose commit outcome was unknown
  60. (for example the connection dropped during the commit); check `drop_reports` for its account and time before replaying
  61. it.
  62. ## Benchmarks
  63. ### Report consumer CPU
  64. `internal/workers/reportwkr/consumerbench_test.go` (build tag `consumerbench`) drives the report consumer through its
  65. real code path: each task is published to a JetStream stream of its own, fetched and handed to
  66. `Worker.ingestPreprocess` (`BenchmarkConsumer/ingest`), which stores it in PostgreSQL and sets its recall key in Redis,
  67. with logging and tracing configured as in production. `BenchmarkConsumer/process` runs `Worker.process` alone. Besides
  68. wall time and allocations it reports, per task, the CPU time of the process (`cpu-ns/op`, user + system from
  69. `getrusage`, which a shared machine disturbs far less than the wall time) and the time spent waiting on PostgreSQL
  70. (`pg-wait-ns/op`, `pg-wait-%`) and Redis.
  71. The tasks (`internal/workers/reportwkr/testdata/consumerbench/tasks.jsonl.gz`) are encoded as the report handler
  72. publishes them, drawn from the stages, servers, times, sources, versions and drop patterns of recent production reports,
  73. with synthetic accounts, IPs and task ids.
  74. It needs a scratch database, Redis and NATS with JetStream. The scratch database holds the schema and the reference
  75. tables of a restore of production (`penguinprod`, only read) and synthetic accounts; the benchmark truncates its
  76. `drop_reports` and `drop_report_extras`:
  77. ```sh
  78. scripts/consumerbench/setup-db.sh # creates penguin_cpubench and the role cpubench on 127.0.0.1:5434
  79. # regenerate the tasks, if needed: CONSUMERBENCH_TASKS, CONSUMERBENCH_BATCH_PERMILLE, CONSUMERBENCH_SEED
  80. CONSUMERBENCH_GENERATE=1 go test -tags consumerbench -run TestConsumerBenchGenerate ./internal/workers/reportwkr/
  81. ```
  82. The defaults point at `penguin_cpubench` on 127.0.0.1:5434, Redis DB 14 on 127.0.0.1:16379 and NATS on 127.0.0.1:4222
  83. (`CONSUMERBENCH_POSTGRES_DSN`, `CONSUMERBENCH_REDIS_URL`, `CONSUMERBENCH_NATS_URL`). Then:
  84. ```sh
  85. # one run
  86. go test -tags consumerbench -run '^$' -bench BenchmarkConsumer -benchtime 2000x ./internal/workers/reportwkr/
  87. # interleaved runs of variants, compared with benchstat: logging to a counting io.Discard (prod), to /dev/null
  88. # (devnull), disabled (logoff); tracing as in production at 10% (prod), off (traceoff), at 100% (trace1); both off (off)
  89. COUNT=10 scripts/consumerbench/run.sh bench prod logoff traceoff off devnull trace1
  90. # CPU and allocation profiles of the ingestion, the top functions and the share of each part of the consumer
  91. scripts/consumerbench/run.sh profile prod
  92. ```
  93. Outputs go to `$OUT` (default a new temporary directory). Each run logs the load average and the queries other sessions
  94. had active on the cluster to `$OUT/runs.log`. The PostgreSQL connection runs over loopback, where a write system call
  95. also does the receiving side's TCP work, so the system calls weigh more than against a remote server.
  96. ## Maintainers
  97. This project has mainly being maintained by the following contributors (in alphabetical order):
  98. - [AlvISsReimu](https://github.com/AlvISsReimu)
  99. - [GalvinGao](https://github.com/GalvinGao)
  100. > The full list of active contributors of the _Penguin Statistics_ project can be found at the [Team Members page](https://penguin-stats.io/about/members) of the website.
  101. ## Contributors
  102. This project is made possible by the following contributors. Contributions are always welcome!
  103. [![Contributors](https://contrib.rocks/image?repo=penguin-statistics/backend-next)](https://github.com/penguin-statistics/backend-next/graphs/contributors)
  104. ## License
  105. [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fpenguin-statistics%2Fbackend-next.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fpenguin-statistics%2Fbackend-next?ref=badge_large)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.