This is a post in a series where I complete every Flare-on challenge. The landing page for all of these posts can be found here
The second challenge from this season built on the first challenge. It was another password entry challenge but with a more complicated password encoding scheme.
First things first I validated what kind of file I was looking at.
λ file very_succes very_succes: PE32 executable (console) Intel 80386, for MS Windows
When running the file I entered some test data to see how it looked to a user.
I switched back to Ghidra to do some static analysis on this binary and found the area of code that looked to handle the password comparison. The first check that jumped out to me was the length check that checked to see if the password was 37 characters long.
Then I found the encoding and matching bulk of the code which I have commented below. This block of code uses a combination of XOR and Bit-wise shifting of the characters to encode each character of the input to match it against the encoded password.
It took me a little bit to see that the SCASB instruction at 0x4010c8 is used to set the zero flag to 1 if the encoded value does not match and jump to a failure condition. Otherwise is set to 0 for success and continues the loop.
I ran the binary using x64dbg to walk through and monitor execution manually setting the Zero Flag to check how the algorithm operated. I also identified the location of the encoded key stored in EDI and copied out that data in hex.
AFAAADEB AEAAECA4 BAAFAEAA 8AC0A7B0 BC9ABAA5 A5BAAFB8 9DB8F9AE 9DABB4BC B6B3909A A8
As with the first challenge in this season I crudely implemented the encoder in python and using brute force was able to successfully generate the key.
encoded = [0xAF, 0xAA, 0xAD, 0xEB, 0xAE, 0xAA, 0xEC, 0xA4, 0xBA, 0xAF, 0xAE, 0xAA, 0x8A, 0xC0, 0xA7, 0xB0, 0xBC, 0x9A, 0xBA, 0xA5, 0xA5, 0xBA, 0xAF, 0xB8, 0x9D, 0xB8, 0xF9, 0xAE, 0x9D, 0xAB, 0xB4, 0xBC, 0xB6, 0xB3, 0x90, 0x9A, 0xA8] result_key = "" def xchg(s1, s2): temp = s1 s1 = s2 s2 = temp return s1, s2 def decoder(text_data): global result_key success_count = 0 bx = 0 dx = 0 key_store = 0 # stack cl = 37 eax = 0x1901c7 for i in text_data: dx = bx dx = dx & 0x3 ah = (eax & 0x0000FF00 > 1) al = (eax & 0x000000FF) dl = (dx & 0x00FF) al = (i ^ al) dl, cl = xchg(dl, cl) ah, cf = ah << cl, ah & 1 al = al + ah + cf ax = al + (ah*0x100) dl, cl = xchg(dl, cl) dx = 0 dl = 0 ax = ax & 0xff output = ax bx = bx + (ax & 0xff) cl = cl - 0x1 if encoded[cl] != output: pass else: result_key += chr(i) success_count += 1 return success_count test =  * 37 for element in range(len(test)): for i in range(0x21,0x7e): test[element] = i succ_coun = decoder(test) if succ_coun < element+1: pass result_key = "" else: print (succ_coun, element, chr(i)) print("Key:", result_key) break print (test) print ("resultkey: \"" + result_key + "\"")