(This is a stripped-down version of the news letters sent out to members of Codeberg e.V. – not member of the non-profit association yet? Consider joining it!)
Dear Codeberg e.V. Members and supporters!
The past months at Codeberg have been intense, and it's overdue to share some updates.
Organisational affairs for e.V.-members: Time for our annual assembly is approaching. While the presidium has not yet agreed on the actual date for our meeting, it will be scheduled within the first half of the year and likely to happen around March or April.
We will elect new cash auditors, and a new presidium must be formed. Please consider to volunteer and run for offices! Please reach out to presidium@codeberg.org to get in touch.
Highlights
Migration to own hardware
Although our infrastructure was not very stable recently, be informed that the migration to our own hardware was successfully completed last year. Since early November, everything is hosted on a physical server owned by Codeberg, housed in a data center in Berlin.
Storage for your projects is on a Ceph cluster, which provides flexibility to seamlessly scale up in the future. The downtime for our migration was kept short, and lasted one hour, but we fixed a small number of bugs the days after. Migration of the mail server is still pending.
Forgejo fork
If you had been involved with Codeberg for a while already, the term "Gitea" is well-known for you, providing the UI codebase for our main service. With recent moves of the new founded for-profit Gitea Ltd, which took over important assets of the project, without community consensus, we considered and decided it's best to support and track a community-based soft-fork instead. We are proud to have helped launching the libre Fork "Forgejo" within our means. It will aim for maximum collaboration with upstream when possible, but reduce risks of "soft-lock-in" and unintended dependencies to commercial ventures, ensuring long-term freedom guarantees.
Collaboration of Codeberg and the Forgejo maintainers has been pleasant and extremely constructive. We look forward to steadily building out this relationship. As aside, last but not least, we appreciate the decision to host Forgejo on Codeberg.
A fair amount of intense work
Unsurprisingly, with a growing number of users migrating to Codeberg, the amount of work is also increasing. Our infrastructure recently experiemnced an unusually high amount of performance issues, even some short downtimes.
For once, we investigated decreased availability and transient errors on our hosted Weblate service, a platform for crowd-translations for your Codeberg projects. These problems were mitigated by providing more RAM for the Weblate container, and the situation appears to have stabilised so far.
We have noticed that our backup solution using BorgBackup did not keep up with the growing storage. Full backup runs took longer than acceptable, and there was a huge amount of disk I/O operations, that slowed down other user-initiated operations on Codeberg. We have since started work on a custom backup concept which consists of a live mirror of your data, and filesystem snapshots will ensure data integrity in the future. To learn more about the project, follow this issue.
The initial run was causing high disk usage again (now done), but we are confident that the concept works out long-term. We are still working on fine-tuning the change detection, and will once more confirm complete data integrity of the mirror over time.
A growing number of crawlers, performing deep scans of the entire website, including all repositories, added additional heavy traffic spikes, degrading user experience for us human beings. We need to improve on traffic shaping and limiting, which is currently unfortunately very tricky at the front end side, as common load balancers (like haproxy we are currently using) are not content-aware, and cannot limit traffic as function of generated CPU and I/O load (and this can greatly vary for GIT repo accesses).
Last but not least, we have spent about € 3.000 to order four 8TB Enterprise SSDs. We learned that the access pattern of growing Git repos with many small disk accesses is much different than for example large file hosting, and thus our HDDs at 12% of their capacity cannot even keep up with the I/O operations. We will install them into our server as soon as possible, further improving bandwidth and access times. Update to newsletter via mail: The SSDs have been installed, and are now being added to the Ceph cluster.
Where we appreciate help
We recently published an honest blog article about "The hardest scaling issue".
If you want to help solving it, we appreciate your helping hand. Most urgently needed are people who can take responsibility of a project independently and coordinate volunteer contributions therein.
For example, you could help set up and maintain Code Search, see this issue. Or, you could join maintenance (code or sysadmin) of Codeberg Pages, Weblate, Woodpecker CI, or help us maintain LXC containers, HAProxy, Ceph filesystem and more. Please let us know!
Our moderation and community issue tracker at Codeberg/Community remains an active place to discuss bugs, features, and community-related needs. If you have some free cycles, please go through the recent issues, help out users, forward bug reports and feature requests to upstream Gitea at GitHub if they aren't reported yet, and consider joining our group of active community maintainers: see Codeberg/Community#571 for details.
Codeberg e.V. has 294 members in total, these are 209 members with active voting rights, 83 supporting members and 2 honorary members.
Your Codeberg e.V.
Friendly reminder: membership in the account group "Members" on Codeberg.org is not automatic (this group enables access to Codeberg e.V.'s internal repos and discussion between Codeberg e.V. members therein). For privacy reasons we add members on request (your membership is visible to other members). If you are not yet in this account group, but would like to join, please send an email to codeberg@codeberg.org and tell us your username. If you need access to Codeberg e.V. report documents without being listed in the account group, please send an email.
--
https://codeberg.org
Codeberg e.V. – Gormannstraße 14 – 10119 Berlin – Germany
Registered at registration court Amtsgericht Charlottenburg VR36929.