(This is a stripped-down version of the news letters sent out to members of Codeberg e.V. – not member of the non-profit association yet? Consider joining it!)
Dear Codeberg e.V. members and supporters!
It is a pleasure for us to provide service to you. You are an awesome community. Although we don't always have the time to engage with every discussions, we appreciate reading your thoughts on Mastodon, our Matrix channels and our issue trackers.
We wish you and your projects all the best for 2025.
A reminder: The annual membership contributions are due. If you have chosen SEPA direct debit as a payment method and your membership fee was not yet collected in January, you now have the last chance to update your payment details or cancel your membership. We will collect the last pending transfers early February. Our bank account has a monthly limit, and although we increased it year-to-year, we still have a few pending transfers.
If you have chosen manual wire transfer, PayPal or another payment method, please send your chosen yearly membership fee now.
If you're interested in learning what we've been up to in the past few months, read on!
Highlights
- Codeberg @ FOSDEM 2025: Meet the team in Brussels and get cool stickers in return!
- Our internal member management is "all new and shiny", enabling faster responses and easier management for us.
- Infrastructure improvements. We have expanded our Ceph storage cluster to three physical machines, reworked our network setup and are slowly clustering services.
- Forgejo is now over two years old!
Codeberg e.V. has 663 members in total, these are 452 members with active voting rights, 204 supporting members and 7 honorary members.
We're going to Brussels!
We are excited to attend FOSDEM in Brussels, Belgium.
Admission is free (as in beer!), and the event is open to everyone. For more information on how to get to FOSDEM, check out FOSDEM's website. You can find Codeberg's and Forgejo's joint stand at the UA building (Level 2).
Throughout the event, we will be wearing a pin with the Codeberg logo, so you'll be able to spot us easily. If you do that, ask us for Codeberg and Forgejo stickers! We will (hopefully) have enough for yourself, your friends, family, colleagues and your hackspace. While we're there, we might (if our time and capacities allow!) also visit other events taking place in Brussels, such as Bytenight, or even hack on some Forgejo code!
In any case, we will be there. You are invited to join our Matrix room #codeberg-fosdem:bubu1.eu (matrix.to) and join us, or invite us to cool things yourself. :)
Member management
By being Codeberg member, you help us keep Codeberg sustainable, and you get to have a say on important decisions affecting its future.
As of today, we have more than 600 members. Member management is an integral part of our day-to-day business, however, as one might understand, it can get complicated very fast. Previously, this required a lot of regular manual effort on our side. From granting people access to the Codeberg-e.V. organization, initiating SEPA direct debit every month by hand (and yes, there have been repeated cases where we simply forgot), responding and tracking to change requests...
(Not yet a member yet? Join us today!)
We used to do all of this using an off-site "database". The database was... actually a CSV file whose changes were tracked using Git, together with a bunch of Python and Bash scripts for extra convenience. This might sound scary to some, but it actually worked very well for the first few years! However, as Codeberg e.V. grew, so did the complexity. For example, we had people join our non-profit as members, leave it, join it again, and, finally, leave it again.
Joining in the first place used to be quite complex, so we overhauled our registration server (join.codeberg.org). Its newer versions had some features that we had wanted for a long time, but they were incompatible with our scripts.
To remedy this technical debt problem, our employee, @ashimokawa, overhauled our member management tooling. It offers us the following benefits:
- Easier verification of new members. Accepting or refusing applications used to take several minutes, but is now is a matter of a simple click.
- Efficient change tracking. It is much easier for us to edit your data on your request, and we do so in a privacy-compliant manner. However, it will take a while for us to fully rely on tihs system, because the law obliges us to keep old records for a few years.
- Extensibility. It is now much easier for us to add new fields to our membership form. For example, it is now possible to associate your Codeberg membership to your account on Codeberg.org.
- Payment tracking. The new system allows us to track due membership fees across the many different payment methods that we accept. Previously, we could only do this for the SEPA direct debit payment method, which only exists in the European Economic Area.
In a nutshell, this means that you get much quicker response times when contacting us about your Codeberg e.V. membership. As far as we are concerned, we get to spend less time on bureaucracy.
Infrastructure
Most of you are probably more excited about computers. Having an employee means that we get to accelerate our infrastructure's expansion. Many of our ideas that were lingering in our minds for years have finally turned into action. Several of them have been completed, or have at least seen significant progress.
Codeberg.org used to host tens of thousands of users and their projects using one, single server. Depending on who you ask, this might come across as impressive, absolutely horrifying, or both. Regardless, we received a hardware donation recently, and documented the lengths that we took to pick it up on Mastodon! We had to fiddle with our servers' hardware and with some bricked power supplies, but, at long last, we deployed two new servers in our datacenter. Codeberg.org's production setup now relies on a three-machine ring setup.
@liske helped us make significant changes to our networking setup to increase its reliability. Previously, our setup even had some bugs that made server reboots require manual intervention. Now, our setup has a private 10G ring connection, which provides us with a high capacity for internal traffic.
Our hosting provider assigned us new network addresses. We agreed to use BGP to realize virtual / floating IP addresses. We will use them to distribute our services across several IPs, and to prepare for transparent failover. Our IP addresses might change at some point starting from February.
For our SSD-backed Ceph storage cluster, we went for the 8TB DC600M model from Kingston. Even if Kingston doesn't provide official instructions for updating this model's firmware on Linux, their support helped us to do so anyway. We had to use a very scary-sounding command (hdparm --fwdownload xx.bin --yes-i-know-what-i-am-doing --please-destroy-my-drive), but, in the end, it all worked out.
We updated our Ceph cluster to the Reef version, and scaled to three nodes (servers). For cost efficiency, we still use a mix of SSD- and HDD-backed storage. However, SSD-backed storage is used for most (if not all!) read operations. If one servers reboots or fails, our cluster will continue working. This redundancy has resulted in much faster reboots for our primary server as well.
Clustering other services
With clustered storage done, we are now looking into clustering other services. We have made successful experiments with database replication using Galera Cluster (MariaDB), and we are about to do a dry run using a snapshot of production data within the next days.
We have not yet decided for a distributed search index for issues. Currently, a local bleve index is used. We consider Meilisearch (and tested it successfully), but it does not allow scaling out of the box. Other options include OpenSearch (which seems quite heavy) or Typesense (which would require adding support in Forgejo).
During the preparations to cluster Forgejo, we have discovered that about 400 GB of Forgejo Actions CI/CD logs and artifacts have not been written to our distributed storage, because these features have been enabled after the initial storage setup (and it didn't matter for a single instance). During a short maintenance downtime of about five minutes, we moved this data to the Ceph storage a few days ago.
Currently, Forgejo is not ready to support high availability or clustering out of the box. We expect that this will require some effort and fixes over the next years. Some caveats exist, but we have a few ideas to work around them for an initial clustered setup. You can follow our progress in a discussion issue.
We hope to launch a second instance of Forgejo for Codeberg.org soon. To prevent race conditions, it will only handle traffic with specific characteristics. Effectively, it will be treated a "read-only" instance.
Given the rise of careless AI crawlers and other bot traffic, DDoS attacks or other resource abuse (such as using Codeberg as a CDN for pirated content or malware), Codeberg often suffers from performance degradation caused by some rather rude "guests". To protect those that rely on Codeberg to develop their projects, we aim to direct "read-only" traffic to a separate instance.
Although this does not fix the underlying causes, nor enable load-balancing or instant failover (yet), this first step should protect the platform against most cases of unexpected downtime, while increasing its availability and performance.
Forgejo
Forgejo is now over two years old! Not only are we proud of all the progress and useful features that Codeberg.org users benefit from, but we are also super happy about the thousands of Forgejo instances hosted in datacenters and Raspberry Pi's worldwide! Forgejo is funded transparently, and part of Codeberg e.V.'s allocated budget goes to Forgejo.
If you're interested in monthly project updates and release notes, check out Forgejo's dedicated blog.
Forgejo's test suites have been improved significantly. We believe that it is one of the few libre, non-profit software projects that implements browser testing to verify that what the user sees is correct.
We also significantly improved the performance of database queries and code search. This helped reduce the load times of some pages on Codeberg.org. However, for those that use Codeberg heavily, many pages (in particular, the Dashboard) can take a long time to load. These remain a high priority.
We have made significant progress for accessibility and usability (for news regarding our inaccessible CAPTCHA, read on). Some forms and pages have been reworked, following accessibility and UX best practices. An important change was the new repo dialog, which has been drastically simplified from the previous rather messy mix of options.
Forgejo's localization (and thus Codeberg) has been greatly improved, both in quantity and quality! Filipino and Lower German ("Plattdüütsch", a minority language spoken mostly in the North of Germany) are now completed translations. If you speak those languages, we hope you appreciate the effort.
However, there are always more things to be done and bugs to be squashed. When we get too busy or need a break, we focus on fixing details as well - such as a misaligned UI button or a sentence that just sounds a bit weird. These changes pile up over time, and we are quite happy to say that we believe that Forgejo's overall quality has been polished a lot over the past few months.
A big thanks to all contributors to Forgejo. In this newsletter, it is impossible to mention everything that was done in 2024 by the more than 700 people who improved Forgejo through bug reports, suggestions, user research, translations, code contributions and many other ways! You are great!
Although the budget allocated by Codeberg e.V. compensates some large and important efforts, it is not enough to compensate the Forgejo maintainers continuously. If you can, consider sending money to them directly on Liberapay (by doing so, you also spare Codeberg with the extra paperwork).
Spam and abuse
Spam is a never-ending story, and not a very pleasant one at that. We'll keep things short by focusing on the good news.
Recently, we have started to deploy basic spam filtering based on usernames. There are some patterns that are obviously spam (such as "casino123" or "Carkeyreplacementtexas"), and this reduced the inrush of spam significantly. This was very successful, so we started using more patterns (such as "service" followed by three numbers).
Unfortunately, this resulted in some false positives as well. To those few whose registration was filtered: Sorry!... But, you should choose a better username next time (just kidding! We're working on it).
Initially, we intended to use this idea as a basis for a built-in proper pattern matching solution in Forgejo. This would also take descriptions into account. However, it seems like the community was faster than us. We are in contact with the maintainer of a new solution called Forgejo Guardian, and removed some thousands of user accounts with its help already.
Over the past few years, this marks the first time where we can say that we have made meaningful progress with our spam fighting efforts. Should everything keep going well, we hope to discontinue the CAPTCHA on our registration page, which is inaccessible to users with limited eye vision.
As far as resource abuse is concerned: We have not managed to finish our plans towards a storage quota yet. There are works in progress that would help us track the quota usage in Forgejo better, and selectively lift that quota for the projects that need it. This would help us prevent abuse, while letting us give everyone what they need.
A broken laptop
In 2022, Codeberg e.V. ordered a StarBook MK VI from StarLabs Systems to provide the executive director with a trusted environment to work on the platform and member data. Due to delays in production, it arrived early 2023.
In the early days, we have had issues with standby, probably related to the open-source coreboot firmware. During firmware upgrades, the StarBook bricked in summer and had to be returned to the UK where it was in service for more than a month. Ultimately, the mainboard was replaced.
Back in February 2024, during FOSDEM, the USB-C port stopped working. It would still offer connectivity over USB2, but charging via PowerDelivery and display output was broken.
Late 2024, charging issues with the DC port arose. The machine got bricked during another attempt to upgrade the firmware. As of today, it does not boot, nor does it show any life signs when we connect it to a charger.
We attempted to recover the firmware using an external programmer with no luck. However, the chip is quite shallow, so we might have simply not managed to connect properly. There is an alternative way of accessing the BIOS chip using an FPC ribbon cable, which StarLab even offers in a debug kit.
However, our motivation to invest time and energy into this device is exhausted and our experience with coreboot and firmware recovering is limited on these modern devices. It would be great if someone could recover the firmware, or, at the very least, use it as a learning opportunity.
If you want to give it a try, we hereby offer you the device to take a look. We can hand it over in Berlin, bring it with us to FOSDEM in Brussel or potentially even figure out shipping with you.
Hosted Forgejo Actions CI/CD
Codeberg is providing hosted CI/CD service using Woodpecker CI for several years now. We require an application prior to using the service to contain abuse, but currently do not have strict limits otherwise. This offer will remain available as our recommended hosted CI/CD.
However, in recent years, a CI/CD solution native to Forgejo (and Gitea back then) was implemented. It is nicely integrated into the UI and partially offers compatibility with the GitHub Actions ecosystem.
Many users already embraced the capabilities of this new system by connecting their own machines as "runners" to Codeberg. Actions is also used as a daily driver within the Forgejo project. It can work pretty well for these use cases.
It was perceived as confusing that the integrated "Actions" UI is not usable out of the box and we wanted to save people from the need to host their own runners for simple workflows, so we went ahead and started offering hosted runners in an "open alpha" phase.
We offer a global runner that currently executes jobs with tight restrictions on runtime, to test the system and avoid running into resource abuse immediately. You can find all information on actions/meta. Unfortunately, this is not yet an adequate CI/CD solution for general use.
First, our setup is not very reliable yet. We have chosen a rootless-podman setup for security reasons, which had some unexpected complications, such as workflows failing when their title contains spaces. And we try to keep all writes of the short-lived containers in memory only to reduce wear of our SSD and thus electronic waste. A few months of heavy CI/CD usage can reliably move an nvme disk to trash. Unfortunately, we haven't yet found a way to keep temporary writes in memory only but persisting container images. Help is appreciated.
Secondly, while Forgejo Actions works pretty well in some cases, it has some caveats. For example, most Actions in the GitHub Actions ecosystem are written in JavaScript and require a runtime (such as NodeJS) inside the container to work properly. Since the "clone" part of the repository is an Action in itself, it is currently not trivial to run workflows without NodeJS.
The GitHub way is to offer one large VM image that has all relevant tools preinstalled, but creating and maintaining this seems out of scope for Codeberg. Instead, we are considering to recreate parts of the Actions ecosystem and dedicate it to Forgejo. Actions can also be created without requiring NodeJS, so we could offer "checkout" or "chore" Actions that run well in other containers.
The Forgejo-specific documentation is pretty bare-bones and many people find the GitHub Actions syntax complicated and unintuitive. Still, we hope that we can bring our offering into a state that allows simple workflows for everyone and complex scenarios for the GitHub / Forgejo Actions fans.
Woodpecker CI
We know that some people are annoyed by breaking changes in Woodpecker that require manual action to keep the project workflows running. Please follow the announcements to be informed of upcoming changes in time.
If you still need help with the migration of the old secret syntax to the slightly different and more verbose new syntax, you might want to read this blog article.
There has been some disagreements within the Woodpecker team recently, which lead to the creation of a soft-fork named "Crow CI". We acknowledge many of the problems that lead to this situation, especially the need for an open and transparent governance for libre software projects. It is not yet decided if Codeberg might eventually switch to the soft fork (which is unfortunately also hosted on GitHub).
Update: We have noticed too late that Crow CI development moved to Codeberg. Feel free to give them a star and contribute on Codeberg.
Your help is appreciated
Codeberg is our collective effort. We are independent from big tech and aim for transparency and democratic governance as much as possible. Codeberg is one of the few non-profit organizations that are not primarily backed by large corporate members, but allow individuals to join and participate in decisions.
If you love what we do and have the financial capacity, consider joining Codeberg e.V. today. Also check out the Contributing Repo and discover teams that are interesting to you.
Thank you for reading! Your Codeberg Public Relations team
--
Codeberg.org
Codeberg e.V. – Arminiusstraße 2 - 4 – 10551 Berlin – Germany
Registered at registration court Amtsgericht Charlottenburg VR36929.