Flare-on 2 – Challenge 2

This is a post in a series where I complete every Flare-on challenge. The landing page for all of these posts can be found here

The second challenge from this season built on the first challenge. It was another password entry challenge but with a more complicated password encoding scheme.

First things first I validated what kind of file I was looking at.

λ file very_succes
very_succes: PE32 executable (console) Intel 80386, for MS Windows

When running the file I entered some test data to see how it looked to a user.

I switched back to Ghidra to do some static analysis on this binary and found the area of code that looked to handle the password comparison. The first check that jumped out to me was the length check that checked to see if the password was 37 characters long.

Then I found the encoding and matching bulk of the code which I have commented below. This block of code uses a combination of XOR and Bit-wise shifting of the characters to encode each character of the input to match it against the encoded password.

It took me a little bit to see that the SCASB instruction at 0x4010c8 is used to set the zero flag to 1 if the encoded value does not match and jump to a failure condition. Otherwise is set to 0 for success and continues the loop.

I ran the binary using x64dbg to walk through and monitor execution manually setting the Zero Flag to check how the algorithm operated. I also identified the location of the encoded key stored in EDI and copied out that data in hex.


As with the first challenge in this season I crudely implemented the encoder in python and using brute force was able to successfully generate the key.

Decoder code

encoded = [0xAF, 0xAA, 0xAD, 0xEB, 0xAE, 0xAA, 0xEC, 0xA4, 0xBA, 0xAF, 0xAE, 0xAA, 0x8A, 0xC0, 0xA7, 0xB0, 0xBC, 0x9A, 0xBA, 0xA5, 0xA5, 0xBA, 0xAF, 0xB8, 0x9D, 0xB8, 0xF9, 0xAE, 0x9D, 0xAB, 0xB4, 0xBC, 0xB6, 0xB3, 0x90, 0x9A, 0xA8]

result_key = ""
def xchg(s1, s2):
    temp = s1
    s1 = s2
    s2 = temp
    return s1, s2

def decoder(text_data):
    global result_key
    success_count = 0
    bx = 0
    dx = 0
    key_store = 0 # stack
    cl = 37
    eax = 0x1901c7

    for i in text_data:
        dx = bx
        dx = dx & 0x3
        ah = (eax & 0x0000FF00 > 1)
        al = (eax & 0x000000FF)

        dl = (dx & 0x00FF)
        al = (i ^ al)
        dl, cl = xchg(dl, cl)
        ah, cf = ah << cl, ah &amp; 1
        al = al + ah + cf
        ax = al + (ah*0x100)
        dl, cl = xchg(dl, cl)

        dx = 0
        dl = 0
        ax = ax &amp; 0xff
        output = ax
        bx = bx + (ax &amp; 0xff)

        cl = cl - 0x1
        if encoded[cl] != output:
            result_key += chr(i)
            success_count += 1
    return success_count

test = [65] * 37

for element in range(len(test)):            
    for i in range(0x21,0x7e):
        test[element] = i

        succ_coun = decoder(test)
        if succ_coun < element+1:
            result_key = ""

            print (succ_coun, element, chr(i))
            print("Key:", result_key)
print (test)
print ("resultkey: \"" + result_key + "\"")

Author: Ben Mason

Technical Architect - Computer Networking - Security - Electronics Hobbyist - Sometimes Photographer - Spaceflight - Cat Enthusiast - HAM KC1GDJ

Leave a Reply

Your email address will not be published. Required fields are marked *