Cutting Server Load by 97% Without Buying a Bigger Server

Case Study: Infrastructure Rescue and Search Modernization (MediaWiki)

Overview

For an educational research wiki, a surge in automated crawler traffic had pushed the site's server past the point of usable performance, while a legacy search tool was leaving most of the archive effectively unfindable. The goal was to bring the server back under control without buying more hardware, and to give visitors a way to actually locate content across an archive built up over nearly two decades.

The Challenge

The site had two separate problems that had arrived at the same time, and the obvious fix for each one carried a cost the organization could not justify. The primary challenges included:

• Capacity Exhaustion: A modest four-core virtual server was running at roughly fourteen times its safe capacity, driven by automated traffic from AI training crawlers. Human visitors experienced extreme slowness, and the site was heading toward outright failure.
• Unfindable Content: The legacy search was limited to basic keyword matching against page titles. Much of the archive exists to help readers working through difficult personal questions; search that shallow meant a visitor arriving with a specific question would often leave without ever reaching the article written for them.
• An Aging Platform: The MediaWiki installation was three versions behind current. Modern search tooling would not run on the old codebase, so the platform had to be brought current before anything else could be built on it.
• Cost Pressure: The conventional answer to both problems was a larger server and a hosted search subscription, which would have meant a permanent increase in the site's monthly operating costs rather than a one-time expense.

The Solution: Staged Stabilization and Search Rebuild

We worked through a three-phase strategy: Stabilize, Upgrade, and Modernize. The sequence was dictated by the problem itself, since a server under this much load cannot safely be upgraded, and a modern search engine cannot be indexed against an outdated codebase.

Key Features & Technical Highlights:

• Stabilize - Multi-Layered Bot Mitigation:
  • Graduated Defense: We implemented a three-layer response to the crawlers, combining published crawler instructions, web-server rate limits applied to specific offending agents, and MediaWiki's "miser mode" to suppress the most CPU-intensive background operations.
  • Selective Throttling: The rate limits were targeted at the named crawlers rather than applied broadly, so that legitimate search engine indexing and human visitors were never caught by the same restrictions.
• Upgrade - Platform Migration:
  • Codebase and Database: We upgraded MediaWiki from 1.43 to 1.46, carrying roughly two dozen extensions and skins forward and completing the associated database migration.
  • Staged Rollout: The entire upgrade was rehearsed against a full backup in a local environment before any change reached the live site. Two extensions proved incompatible with the new release and were repaired during that rehearsal, so the problems were resolved before visitors could encounter them.
• Modernize - Production Search Engine:
  • Wikipedia-Grade Tooling: We deployed CirrusSearch paired with OpenSearch—the same search architecture Wikipedia itself runs on—and installed and secured it on the existing server rather than subscribing to a hosted search service.
  • Full Index and Live Updates: We built a complete index of the entire archive and configured automatic indexing, so that new and edited pages become searchable in real time.
  • Search Capability: The archive gained relevance ranking, full-text search across the body of every article, phrase searching, and support for advanced query operators.

The Results

• 97% Reduction in Server Load: The server's load average dropped from a critical 57.3 to a healthy 1.5 within minutes of the mitigation going live, returning the site to normal responsiveness.
• Zero Added Infrastructure Cost: Both the performance recovery and the new search engine were delivered through software optimization on the existing server, avoiding a hardware upgrade and a hosted search subscription.
• A Fully Searchable Archive: The entire archive is now indexed and reachable through relevance-ranked full-text search, turning a collection that visitors had to already know their way around into one they can genuinely explore.
• Latent Risk Mitigation: By rehearsing the platform upgrade against a backup first, we found and repaired the incompatible extensions in a controlled environment. The same upgrade applied directly to the live site would have broken visible functionality with no straightforward path back.
• Future-Ready Infrastructure: The site is now positioned to support a planned AI chatbot integration, since a properly indexed, relevance-ranked search layer is exactly the high-quality data source such a system depends on.
• Continuity Value: Because crawler behavior changes constantly and MediaWiki continues to release, both halves of this work will drift without attention. Ongoing maintenance keeps the traffic defense tuned and the platform current, which means the next crawler surge is a routine adjustment rather than another emergency.

Summary

By sequencing the work deliberately, we resolved an immediate performance crisis and then used the stabilized platform as the foundation for a substantial capability upgrade. The archive now runs comfortably on the hardware it already had, its full collection is searchable to a standard visitors will recognize from Wikipedia, and the organization avoided the recurring costs the obvious solutions would have carried. What began as an emergency response became a modernization that leaves the site stable today and ready for whatever the organization wants to build on it next.