I guess it's not really 'hardest' but as far as 'craziest thing discovered while fixing bugs' is this:

I was working for some academic testing software trying to find why the system had slowed to a crawl. I had some test accounts for admins and teachers. During this process I discovered that the passwords and all personal information was stored in the database in plain text. Okay that's pretty bad, but it gets worse.

I decide to follow the rabbit hole and here is what I discover: A user can 'change their password' and it will display their previous password in plain text. They can 'change' to a 'new' password, by using the exact same password they had previously used. Additionally, they could choose 'forgot my password' and it would show you a pretty standard security question and answer page. Only the questions and answers were user submitted, and could be left black. So you could choose any random user that you had an email for, click 'forgot my password', and security question would pop up that was likely blank, the answer field was also likely blank, it would display their old password and you could then change to the same password they were always using.

The sad thing is that when we brought this up, "We wanted users to be able to know what they were currently using so that they didn't use the same password when changing it. We also wanted to allow them to choose a custom security question and answer as we feel this is better than the regular security questions other sites used".

Essentially, this is the best "It's not a bug, it's a feature" I've ever seen in real life.

I just seriously killed like, 35 cucumber beetles.

Seriously though, I found an Exchange bug with Netbackup Infostore agent. I can't remember precisely the details but it had to do with Transaction logs of a specific notation being truncated upon restore. I was able to detect and describe the bug to Microsoft 3rd level and they issued a patch pretty damn quickly. Also developed an interim work around which involved a specific hack of the naming convention of the file to restore. It was pretty good I was able to detect this through routine maintenance, did all the detective work, and fixed it for my company before microsoft! Super bonus for not running in to this Exchange bug in a DR situation due to good practice!

Sooo many awful memories. Among them - IE's horrible stylesheet limits (exhibit A) (exhibit B).

You're doing your best to keep your massive web app modular and you're using different stylesheets for all kinds of controls and modules. One day, some of your styling isn't working. But it's different pieces in different areas that aren't being applied. There's no rhyme or reason to it. Finally you start just mashing styles from one file into another and... behold, it works. So... then you decide aww man, this bug is a nightmare. We have sooo many files. Okay, we'll get around it by just mashing all of them together and serving up a single CSS file. NOPE, there's a limit to the number of rules you can have in a file... and file size too. It's SO fun. So, then you get to spend your time guessing and checking and tinkering until you've found the magic balance between number of files, number of rules, and size of files that makes IE happy.

...and people wonder why Devs hate IE so much that Microsoft is changing the name.

Oh, boy. This was just last week.

The way our software synchronizes across machines is by serializing objects and multicasting them out. Other machines subscribe to the multicasts, tell the other machines which objects they want, and deserialize the objects as they come in. I ran into a particularly weird issue with doing this over an Ubiquiti NanoBridge M5. I could see all multicast data using Wireshark on all machines, I could ping all machines from all others, the network looked good as far as I could tell. But for some reason, our software on the other side of the NanoBridge from the main server wasn't actually picking up the data. Wireshark said it was there, but it just wasn't getting picked up.

Turned out to be a configuration issue with the bridge. For some reason, even though I could ping the main machine, it wasn't getting the callback to actually push out the data to the machines across the bridge. A one-way block, for some reason. Luckily we were able to Google the correct configuration, but that was 6 hours of banging my head on the wall.

If it's intermittent and unpredictable its always a race condition.

once when I was little there was a bug with some sort of shell that was mad hard, to smash it I had to use a rock since it was impossible to do with your finger. your armor doesn't do jack against rock you little bitch

Not hard, but definitely an unexpected one.

One day I had to adjust some GUI element, just align it better with other stuff, and out of the blue some important feature that had passed all tests and which I had personally used the day before, stopped working. Out of the blue.

Turns out it was a function that showed a dialog and was supposed to return an integer based on what the user selected, but instead of returning something it just flowed out the function. There was a warning about that which was obviously ignored. But for at least 6 months, the code magically happened to return the value I intended it to... until I changed the UI, it returned garbage and things crashed down the road.

I was working on a project that injected profiling and debugging instrumentation into existing C or C++ code. In order to test the project, I was using Google's open source Javascript engine, V8, as it's a reasonably large, reasonably complex C++ code-base. On its own, V8 worked fine, but with the additional instrumentation added by the project I was working on, there was a very strange looking crash.

After about 2 days of stepping through assembly code in a debugger, I figured out what was going wrong. The V8 engine was JIT compiling Javascript to x64 machine code then executing that on the fly. The general pattern was that any registers clobbered by a JIT compiled block were preserved on the stack before execution, then restored afterwards. In one particular fragment, the rdi register was being used without being properly preserved (the rsi register was being accidentally preserved instead). The existing V8 code just happened to work despite this register being accidentally clobbered, but the instrumentation that my project injected was using this register both before and after the execution of the JIT compiled block. Since it was being accidentally clobbered by V8, my instrumentation was crashing! In short, the bug was in V8 itself but was hidden and only became apparent with the addition of my instrumentation.

It was registered as issue 2238 and was fixed here!

At the end of the day, I was quite pleased that it wasn't my bug, but that was certainly a rough debugging session!

So there I was, building a top down maze game in Java. I have the controls working, the sprites moved, and you could finish the game. In short, everything worked. Aside from a small tiny detail. After 600 pixels down, the player sprite would disappear. I never did solve it, so what I did was scale the entire thing so everything fit under 600 pixels.