Microdevops

Why backups stored on the same server can’t always save you

When Backups Fail: A Lesson in Real-World DevOps Why backups stored on the same server can’t always save you. In the world of DevOps, we often talk about backup strategies. Everyone nods, everyone agrees, and most teams say: “Yes, we have backups.” But let’s talk about a real incident we’ve seen in practice. A production server, hosting ...

We noticed significant server overload after migrating cron jobs to run inside Docker containers. The number and logic of jobs hadn’t changed – only the method of ex*****on had. This triggered a deeper investigation into the cause.

Server Overload Caused by Docker-based Cron Jobs Real-world DevOps stories and lessons learned from our work – practical case studies highlighting challenges, solutions, and outcomes.

Hetzner, one of the leading dedicated server providers, has announced a large-scale mainboard replacement for several server models (AX42, AX52, and AX102) due to a design flaw. This replacement process, begun in February 2025, will take up to a year and affect numerous dedicated servers across multiple data centers.
https://microdevops.com/blog/technical-updates/hetzners-mainboard-replacement-microdevops-response/

Struggling to manage your passwords? Learn the risks of poor password habits, how to organize your credentials securely, and practical steps to prevent account loss, security breaches, and unnecessary stress.

How to Organize Your Logins and Passwords

Microdevops is proud to announce that it has officially secured trademark registration in the European Union. This milestone not only strengthens the company’s brand identity but also underscores its commitment to providing innovative DevOps solutions to businesses across Europe.

Microdevops Secures Trademark Registration in Europe Microdevops is proud to announce that it has officially secured trademark registration in the European Union. This milestone not only strengthens the company’s brand identity but also underscores its commitment to providing innovative DevOps solutions to businesses across Europe. The trademark reg...

Microdevops Expands to Balkans

We’re thrilled to share some big news: Microdevops now has an office in Croatia! As part of our growth journey, we’ve officially registered a Microdevops d.o.o. (LLC) in this beautiful country.

Our focus remains on delivering top-tier DevOps services. While our infrastructure continues to be primarily based in Europe, with servers in Germany and Finland, this expansion allows us to enhance our capabilities and broaden our global reach.

Our work mode remains remote-first. We’re committed to providing you with the care and attention you need through our dedicated support chat—all while adhering to your Service Level Agreement (SLA).

Why Croatia? Croatia offers a growing tech ecosystem and a strategic location in Europe, making it a perfect choice for a company like ours that values innovation, connectivity, and diversity.

What’s next? Under the leadership of Yevhen Antonov and backed by a skilled team, we’ll continue to provide:

Reliable infrastructure and operations management.

Support for your growing technical projects, anywhere in the world.

Let’s grow together. If you have any questions or need assistance, our team is always here to help. Together, we’re redefining how businesses achieve efficiency and scalability in their IT operations.

For more details you are welcome to visit our website https://microdevops.com/microdevops-expands-to-balkans

Troubles, Reasons, Use Case and Approximate Costs of Website Troubleshooting. How to Troubleshoot a Website with Microdevops.

So, you own or manage a website like Online Shop or Blog and looking how to troubleshoot a website.

Troubles:
- 400 Bad Request
- 413 Request Entity Too Large
- This page isn’t working (HTTP Error 500)
- 502 Bad Gateway
- Gateway time-out (Error code 504)
- Web server is down (Error code 521)
- Your connection is not private (NET::ERR_CERT_AUTHORITY_INVALID, NET::ERR_CERT_INVALID)
- This site has been reported unsafe
- Redirected you too many times (ERR_TOO_MANY_REDIRECTS)

The errors may vary but the consequences are usually the same:
- Loss in SEO
- Removal from the SERP
- Lost orders
- Loss of the money spent on ads
- Lost profit
Sometimes you don’t have critical errors but the website isn’t loading quick enough, and that leads to, again, lower SERP positions, visitors close the page because they don’t to wait.

We Can Help
We believe that every website deserves great operation and we know what to do to make it happen and how to troubleshoot a website.
We will discuss your website issues, we provide 2 hours consultation for free.

For more information you are welcome to visit our webpage https://microdevops.com/website-troubleshooting/ and do not hesitate to contact us via DM.

WAF in CF Pro and again better monitoring

We have developed for ourselves an optimal set of WAF rules, which can be set on the CloudFlare Pro plan to protect a typical online store:
1. Allow (Skip all components) AS15169 (Google), as practice has shown, for example, Google Merchant is not included in Known Bots, so it is better to open all Google
2. Allow Known Bots – for normal indexing, SEO
3. Allow IP addresses of partners: catalogs, banks, etc
4. Allow back office server IP addresses
5. Allow monitoring IP addresses
6. Allow access to custom URIs with feeds (eg .xml)
7. Block the admin pages for all IPs except the trusted list
8. Block all countries except the ones you need, for example for Europe: (not ip.geoip.country in {"AL" "AT" "BE" "BA" "HR" "CZ" "DK" "EE" "FI" "FR" "DE" "GR" "HU" "IE" "IT" "LV" "LT" "MK" "MD" "ME" "NL" "NO" "PL" "PT" "RO" "RS" "SK" "SI" "ES" "SE" "UA" "GB" "BG"})
9. It is also worth checking that cron requests from the server bypass CF, for example like this: /usr/bin/curl --resolve example.com:443:127.0.0.1

Added standard pillars that regulate the behavior of segfault, coredump monitoring: lower priority or disable, and added exceptions to typical sources of noise in monitoring.

Updated GitLab for clients to version 16.3.4.

Fixed the bug (https://github.com/microdevops-com/microdevops-utils/commit/ff882a555e3d92e8b0fa33f4178da5d30ec28a81) in the heartbeat service, which led to unstable operation of this service.

Made several standard pillars that allow you to connect to a port from the Internet, in which an exception for monitoring open ports is made at the same time, for example this one (https://github.com/microdevops-com/microdevops-formula/blob/master/pillar/ufw/ftp_passive_from_any.sls).

Fixed state error (https://github.com/microdevops-com/microdevops-formula/commit/5f8c1c36abca634716aced09a00f56bd5b74c573) cmd_check_alert, which slowed down the ex*****on of this state and led to short-term monitoring errors in some situations.

For more details you are welcome to visit our webpage https://microdevops.com/waf-in-cf-pro-and-again-better-monitoring/

Tried Google Ads, our stack features updates

Hey-hey, we are 7 years old already! Time flies :)) We’ve been trying to register our business on Google maps for more than a year, and finally – BD present )) we managed it! It turned out this option for entrepreneurs from Ukraine was blocked because of war.

We decided to give a try to use Google Ads to advertise our site.
We, mostly engineers, do not understand marketing through Ads very well :), we were constantly told not to do Ads on our own, but we tried. We’ve got some interesting observations:
Dealing with the Ads service itself and getting clicks on the site turned out to be much easier than expected. And it is quite possible to do it independently without the involvement of third-party organizations.

The effectiveness of Ads for attracting clicks/impressions seems to be much higher than SEO. At least for our business.

But we are grateful to the existence of word-of-mouth marketing, thanks to which the business develops even without an active external search.

We registered the domain https://micrdevops.com.ua, mainly so that it is not registered by unkind people :). We also applied for trademark registration in Ukraine and the EU, the USA is next in line.

In our main backup utility rsnapshot_backup (https://github.com/microdevops-com/microdevops-formula/blob/master/rsnapshot_backup/pillar.example) for a long time there was a shortcoming that local backups, including copying multi-gigabyte database dumps, were performed via ssh on localhost. This behavior has finally been changed and rsync of local backups is now performed without ssh, which is better in terms of performance.

For our clients, we upgraded their gitlabs to version 16.2.

We received an interesting case with one of the new clients: the client used many virtual machines with an Admin panel, in which a vulnerability was found by hackers and malicious code was injected into the sites through it. Because of that these servers were blocked in Hetzner. When the client turned to us: we found a way to get the data from the servers, clean malicious injections and upload sites to a new separate dedicated server. Thus save the search reputation of these sites. Next, since our ideology of Infrastructure as Code is not compatible with any panels, but the client constantly and massively generates many sites, we implemented the Client – IaC connection for it. The client gives us a list and parameters of the sites to be created on its endpoint, and the deployment pipeline takes this list and deploys the sites (static, WordPress) on the server using IaC.

For more details you are welcome to visit our website https://microdevops.com/tried-google-ads-our-stack-features-updates/

Polishing of tools

With a small fix (https://github.com/microdevops-com/microdevops-utils/commit/6170aaa7ca19fe288d243967352838ac837150c7), we’ve augmented the rsnapshot_backup (https://github.com/microdevops-com/microdevops-formula/blob/master/rsnapshot_backup/pillar.example) with the ability to pipe a dump using mysqldump_args, through, for example, pv. This allows you to slow down the dump, thereby more delicately loading the servers with large dumps.

We documented the client parameters (https://github.com/microdevops-com/accounting/blob/master/clients/example.yaml) in more detail in our system of accounting and management of clients, servers, and billing (https://github.com/microdevops-com/accounting/).

In the same system, errors in the calculation (https://github.com/microdevops-com/accounting/commit/e1c629ad56851384b2dc93ef84b932d3e77b25d6) of the coefficient of the monthly payment for the server have been corrected.

Added the docker builder prune command to the gitlab-runner cleanup script installed via microdevops-formula (https://github.com/microdevops-com/microdevops-formula/tree/master/gitlab-runner). Without this command, over time disk space would become clogged with docker builders caches.

Replaced (https://github.com/microdevops-com/microdevops-utils/commit/f450862951900f0b48ce8799f6ff0dd8b66c4342) the default severity level in cmd_check_alert with critical level. For example, when the specified check command is not found at all, then bash issues exit code 127, and then it turned out that in some cases we expected a critical alert, but received a major severity level. The updated logic is more correct.

Added state to the formula to install collabora (https://github.com/microdevops-com/microdevops-formula/blob/master/collabora/pillar.example).

Rocketchat (https://github.com/microdevops-com/microdevops-formula/blob/master/rocketchat/pillar.example) was also added there.

Updated pillar.example and pillar.example.wordpress. These reference examples are now in line with the practices we use in real life lately, the changes are related to the site file folder hierarchy.

We caught an interesting bug on the new Hetzner AX101 servers, which changes the name of the Linux network interface after a reboot. Therefore, we updated the autonomous logging utility bulk_log for logging about all system interfaces (which made it possible to detect this bug) and updated the bootstrap templates for netplan and udev, which allows us to fix this problem.

And finally, a trifle, but nice: our client, the site klopotenko.com, placed a footer on their site: Server support from Microdevops. It’s very nice when clients do this because we, cats, doing our job great 🙂 .

For more details you are welcome to visit our website https://microdevops.com/