2025-Feb-22: The DB maintenance at my-place.social is complete

2025-Feb-22: The DB maintenance at my-place.social is complete. MariaDB is now running with the jemalloc memory manager instead of the default MALLOC memory manager. Quick tests show improved performance, but we'll have to see.

my-place.social users, please let me know if you notice any changes, good and bad. I am very interested in knowing if there's a noticeable impact on user experience and if the effort was worth it.

Thanks!

Jerry

#myplacesocial

About 1 hour downtime today on my-place.social 22-Feb-2025

Today, 22-Feb-2025, I plan to take my-place.social down for about 1 hour beginning at around 9:15 AM EST (14:15 UTC) to change the MariaDB memory manager to, I hope, prevent the stalls and other issues I've seen on this server. Friendica works the database quite hard, and the default memory manager cannot handle it.

I've never done this before, but if it all goes well, it should need less than 1 hour. Most of the time will be used for making cold backups of the server and the database.

Getting DB errors for 2 missing tables

!Friendica Support
Hi Friendica support. Looking at my logs this morning, I notice some of these errors:

Table 'friendicadb.post-user-view' doesn't exist
Table 'friendicadb.post-view' doesn't exist

Does anyone know why they aren't there? If they should be there? How to add these tables?

Thanks.

How do you get updates if my-place.social is down?

Members of #Friendica server my-place.social:

If the server is down, how can you get status updates?
Members of #Friendica server my-place.social:

If the server is down, how can you get status updates?

Answer 1: If you are active on the "Threadiverse" (ActivityPub servers that are #Reddit alternatives like #Lemmy, #Piefed, and #MBIN), then please join
!my-place.social
from one of these applications. Or follow @my-place.social from a different ActivityPub application, like Mastodon, Sharkey, etc. The posts will go to your timeline.

If you follow @my-place.social from Friendica, Friendica will set it up as a "Group" that you can visit, read, and post to/from. You won't lose messages because they scrolled away on your timeline. They'll be waiting for you to read in the Friendica group. But, this won't help if you follow it on my-place.social and the server is down.

Did you know that Friendica groups is actually built-in #Threadiverse capability? You can use Friendica instead of Lemmy, Piefed, and MBIN to participate in and create communities/magazines. If you follow a community/magazine, Friendica automatically creates it as a group. You can also follow Friendica groups from Lemmy/Piefed/MBIN. Friendica is actually another Threadiverse-capable application, but is often not mentioned.

Don't have a Threadiverse account? Try setting one up at feddit.online

Answer 2: Join the #Matrix room: matrix.to/#/#my-place.social:s…

If you're comfortable with #Discord, you'll find the Matrix user experience to be similar. Matrix is basically a distributed decentralized version of Discord with servers run by volunteers instead of a corporation.

Don't have a Matrix account? You can set one up at secure-channel.net which has information on signing up and where to get apps. Matrix is accessible from every OS and from the web.

If I'm fighting a server issue, it makes me feel better knowing that I have a way to communicate information to people. Joining these also provides a way for you to discuss something related. Please consider joining one of these two solutions.

Um. If you create "servers" on Discord, consider this:

Discord collects a significant amount of user data, including messages, IP addresses, and usage habits for targeted advertising and other purposes. Matrix and #Jabber/#XMPP does not.

Discord is closed-source; Discord code cannot be independently audited for security vulnerabilities or backdoors. Matrix and Jabber are open-source, allowing for community review and improvement.

Discord is centralized. Everything flows through only their servers. They own your data and your accounts. Matrix and Jabber/XMPP are decentralized.

Discord is another billionaire owned social media application, so you are not important to them. They will take from you whatever they can.

So, you might consider using Matrix or Jabber/XMPP in place of Discord.

#myplacesocial

Where was my-place.social? What happened?

I am so sorry for the very long downtime that my-place.social had this weekend. In over 2 years of running public servers, there has never been such a long outage.

Note that it's going to take the server some time to catch up on missed messages. Also, the database has been restored to about 4 AM EST (9 UTC) from January 4th so there may be roughly 4 hours of information missing from the database.

Here is what happened. Here is what I'm going to do in the future to make sure this doesn't happen again.

WHAT HAPPENED?

I made a snapshot of the instance and backed up the database. I ran the 6 command lines to do the Friendica update and I started Friendica. Instantly, the entire database became corrupted. Just about every table and every index was instantly corrupted.

It was so bad that MariaDB stopped running and refused to start. Why is this unfortunate?

It's impossible to look at, attempt repairs, or even restore from a backup if the database engine refuses to run. I did everything that I could to get back in. To do it, I had to discard indexes and other DB structures. I was hoping I could then find the tables and indexes that were broken, and repair them. But, eventually, I had to give up. Things were hopelessly broken. MariaDB just refused to run.

I had to wipe out every trace of the database software, configuration, and the databases itself and reinstall the database from scratch, as if bringing up Friendica for the first time. I'm not a MariaDB expert. I eventually was able to do it, but a lot went wrong before I figured out everything that needed to do for the reset.

Fortunately, I keep very detailed notes about every server I create, so I was able to go back and create the database, and configure it, the way it needs to be done for Friendica.

Once the database engine came back up, and I had an empty working database, I began to restore from the last backup. I had no idea it would take 11 hours to restore a 36GB database. It restores by running SQL insert commands for every record in every table. Maybe there were a million commands? No idea.

WHAT CAUSED IT?

Seems something related to the upgrade, but perhaps a coincidence? I won't update again until I investigate. At this time, I don't know.

WHAT ABOUT THE FUTURE?

I kept extremely detailed notes about how to remove and restore the database engine, should I ever have this issue again. It will take under an hour next time. I also added another 50GB drive to the computer. Before doing an update, I will take an image backup of the database so if the database is damaged, I won't need to do a half-day restore. I can copy the image back in less than 5 minutes from the backup drive. This will also prevent any data loss.

I suggest everyone follow @my-place.social from Friendica, Mastodon, etc., or join the !my-place.social community in PieFed, Lemmy, or MBIN. I post updates there. You can open an account on feddit.online and join the community from there so you can reach it, should my-place.social be down.

Again, I am so sorry for this outage. I haven't had an opportunity to really play around to make sure everything is working. I hope all is well. Please let me know if you notice any issues. This was an extremely stressful weekend.

Jerry

#myplacesocial

Issues with Wasabi S3

#Wasabi who we use for image store is having problems with their S3 service. I believe it's having a big impact on this server. Not only are some images blurry, but I think some functions are failing because of the errors. I also see slowdowns, probably owing to timeouts.

Wasabi status can be seen at status.wasabi.com

Wasabi issues impacting images and video files

#Wasabi is reporting a problem in their us-east-1 region which is what is used on this server for storing images and videos.

I did notice problems with image issues earlier today on the server.

If you see issues with media and video, this is likely the reason.

Sep 16, 11:53 UTC
Investigating - We are currently investigating an increase in 500 level HTTP responses on customer traffic to the us-east-1 region