(This is a stripped-down version of the news letters sent out to members of Codeberg e.V. – not member of the non-profit association yet? Consider joining it!)
Dear Codeberg e.V. Members and supporters!
Let's take some time to look at what we have achieved in the past months, and what is coming up next.
Inside Codeberg e.V.
News from the non-profit association in short (legal members received a more detailed write-up via email):
- there was an extremely long but productive annual assembly at the end of May
- we welcome our five new honorary members aboard, thank you for your extraordinary contributions to Codeberg!
- a new presidium was formed and decided to publish all decisions to Codeberg members internally
Codeberg in the media
In the past months, Codeberg has received some media coverage. Listen to Otto talking about Codeberg in the Sustain OSS podcast or read what the German Windows-specific magazine Dr. Windows wrote about us.
Also, there are recurrent talks about Codeberg on the social networks, we are mostly aware of Mastodon, and we thank everyone who recommends Codeberg to colleagues and friends.
Infrastructure
Keeping the services of Codeberg running is always a time-consuming work. We faced several downtimes, sometimes even lasting for hours. The major cause and trigger are identified. We'll quickly try to explain the situation (feel free to skip if you are not interested in technical details).
The cause is within our Ceph cluster. When we moved our data from virtual servers to our own hardware, we decided to bootstrap a small and degraded Ceph cluster, allowing for flexible expansion at a later stage without the need for another lengthy storage migration.
However, the downtime we faced exactly because of this premature decision certainty exceeds the expected second migration downtime we wanted to avoid. In retrospective, the decision was wrong. If you deploy heavy technology to deal with availability, you require staff to take care of it. Since we haven't had the experience, we should have chosen a boring filesystem that only fails when the hardware fails, not because of complex setup issues.
When Codeberg is hit by extreme traffic, our Ceph cluster gets slow and clanky, requests stack up, and results in a complete freeze of our machine over time. The good news: We have identified a plan on how to fix this in August, and we will be working on making things better. We also plan to add a second server to the datacenter. If you reside in Berlin in September, consider to join the task!
The trigger is different: Our degraded Ceph cluster, although not the most resilient, runs smoothly, and did so for several months. Normal traffic to it is in the range of 0.5 - 1.5k operations/second, with rare spikes to 2.5k (CEST-evenings, big repo pushes). But sometimes, often during the CEST-mornings, we have seen this traffic rise to more than ten times the normal values.
We have now restricted several IP ranges and a lot of individual IP addresses that appear to do excessive crawling of big repositories during this time. They are not throttled via normal rate limiting, because their rate is well below our thresholds, but they are doing very heavy requests. Running up to 1000 simultaneous Git jobs on large Linux and BSD kernel trees is simply too much for our current setup, especially because a single GET request can amplify to Git operations that do hundreds of I/O operations. We had no luck identifying who is beyond these IP addresses (e.g. search engines).
Community & Events
Codeberg is a volunteer-driven effort. Our diversity of contributors is the most important aspect of Codeberg's awesomeness. With the project growing, the core team has not kept pace, so improving the relationship and onboarding processes has highest priority.
Two initiatives have started recently: For one, we have scheduled regular community meetings, and hereby also invite all Codeberg members to join us. For both technical and non-technical work, we meet once a month each, resulting in two events per month. The next dates are 07/09/2023, 20:00 CEST about organizing the contributing parts, and 21/09/2023, 20:00 CEST with technical topics. All details and upcoming events can be found on the new Calendar.
Additionally, we are building a coordination place for Codeberg Contributors. https://codeberg.org/Codeberg/Contributing is becoming the place for an overview of projects and their maintainers, coordinating teams, introducing contributors and more. It is still rough around the edges, but if you are excited to get involved with Codeberg, consider creating an issue to introduce yourself and explore the projects that are already there.
Community Spotlight: Minigames
Do you sometimes need a small break from work?
What about playing a small round of minigames that others develop on Codeberg?
We can really recommend Librerama,
a collection of simple, but fast-paced minigames,
written by Yeldham using the Godot engine.
You can download it for many platforms or directly play it in your browser.
Check it out!
If you prefer retro racing games,
check out "What the road brings" from Captain4LK.
The version on itch is also directly playable in your web browser.
Check it out
And Huitsi's "Parallel Overhead" version can also refresh your mind during a break,
and is also directly playable inside your web browser.
Check it out
We hope that you have some fun trying them out :)
Up Next
Internal communication is sometimes a little chaotic. There are chatgroups, and issue trackers, but we sometimes still find ourselves copying information across various places to keep everyone in the loop. Some members are currently working on a ticketing system to streamline communication, both internally and externally. It will finally also allow to provide efficient end-user-support, we currently simply can't answer all the requests we receive. You can follow the project via Codeberg-e.V./Ticketing.
Thank you for supporting our mission, and for reading. We are looking forward to meeting you soon during our regular meetings.
Kind Regards Your Codeberg e.V.
Codeberg e.V. has 379 members in total, these are 265 members with active voting rights, 107 supporting members and 7 honorary members.
Friendly reminder: membership in the account group "Members" on Codeberg.org is not automatic (this group enables access to Codeberg e.V.'s internal repos and discussion between Codeberg e.V. members therein). For privacy reasons we add members on request (your membership is visible to other members). If you are not yet in this account group, but would like to join, please send an email to codeberg@codeberg.org and tell us your username. If you need access to Codeberg e.V. report documents without being listed in the account group, please send an email.
--
https://codeberg.org
Codeberg e.V. – Arminiusstraße 2 - 4 – 10551 Berlin – Germany
Registered at registration court Amtsgericht Charlottenburg VR36929.