Skip to content

Shellcoding

Common

Platform-neutral material (CPU architecture, general technique).

Register Layout

Each 64-bit register can be accessed at smaller widths. Writing to a 32-bit sub-register (e.g. eax) zero-extends into the full 64-bit register; writing to 16-bit or 8-bit sub-registers leaves the upper bits unchanged.

64-bit 32-bit 16-bit 8-bit high 8-bit low
rax eax ax ah al
rbx ebx bx bh bl
rcx ecx cx ch cl
rdx edx dx dh dl
rsi esi si - sil
rdi edi di - dil
rbp ebp bp - bpl
rsp esp sp - spl
r8 r8d r8w - r8b
r9 r9d r9w - r9b
r10 r10d r10w - r10b
r11 r11d r11w - r11b
r12 r12d r12w - r12b
r13 r13d r13w - r13b
r14 r14d r14w - r14b
r15 r15d r15w - r15b
  • al/ax/eax are the low 8/16/32 bits of rax; ah is bits 8-15.
  • Legacy high-byte regs (ah, bh, ch, dh) only exist for the original four registers and can't be mixed with REX-prefixed operands.
  • Prefer the smallest width that holds your value. For example, mov al, 59 avoids null bytes that mov rax, 59 would introduce.

Linux

Syscalls

Arch Instruction Number Args Return
x86-64 syscall rax rdi, rsi, rdx, r10, r8, r9 rax
x86 (32-bit) int 0x80 eax ebx, ecx, edx, esi, edi, ebp eax
ARM64 svc #0 x8 x0-x5 x0
ARM32 svc 0 r7 r0-r5 r0
  • Assemble .asm: gcc -nostdlib -static shellcode.s -o shellcode
  • Extract shellcode: objcopy --dump-section .text=shellcode-raw shellcode
  • Analyze shellcode: objdump -M intel -d shellcode

  • Shellcode Loader for debugging

  • Run: cat shellcode-raw | ./tester
page = mmap(0x1337000, 0x100, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, 0,0);
read(0, page, 0x1000);
((void(*)())page)();

Filter Bypass

  • Write bytes and patch during execution to achieve syscall
# translates to code 0f 05 => syscall
inc BYTE PTR [rip+6]
inc BYTE PTR [rip+1]
.byte 0x0e
.byte 0x04
  • Long NOP sleds
.rept 0x1000
nop
.endr

Windows

Same CPU (see Register Layout), but Windows differs from Linux in how functions and the kernel are called.

Calling convention (Microsoft x64)

Purpose Registers / location
Integer args rcx, rdx, r8, r9, then stack (right-to-left)
Float args xmm0–xmm3
Return rax
Shadow space caller reserves 32 bytes on stack before every call
Alignment rsp 16-byte aligned at the call site
Callee-saved rbx, rbp, rdi, rsi, rsp, r12–r15
  • The 32-byte shadow space (a.k.a. home space) must exist even if the callee takes ≤4 args: sub rsp, 0x28 is the usual prologue (0x20 shadow + 8 to re-align after the return address).

Calling conventions (x86 / 32-bit)

Unlike x64 (one unified convention), 32-bit Windows uses several. The convention determines arg passing and who cleans the stack.

Convention Args Stack cleanup Used by
stdcall all on stack, right-to-left callee (ret N) most WinAPI functions
cdecl all on stack, right-to-left caller C runtime, varargs (wsprintf)
fastcall ecx, edx, then stack (right-to-left) callee some internal/perf functions
thiscall ecx = this, rest on stack callee C++ non-static methods
  • Return value in eax (edx:eax for 64-bit values).
  • Callee-saved: ebx, esi, edi, ebp.
  • No shadow space (that's an x64 thing).
  • The caller/callee cleanup split matters for ROP/stack-pivot math: a stdcall callee pops its own args with ret N, shifting esp for you.

Syscalls vs. WinAPI

  • Linux syscall numbers are a stable ABI; Windows Nt*/Zw* syscall numbers are undocumented and change between OS builds. Hardcoding them is fragile.
  • Real shellcode therefore calls documented WinAPI functions (WinExec, CreateProcessA, LoadLibraryA, VirtualAlloc, …) instead of issuing raw syscall instructions.
  • Direct/indirect syscalls (resolving the number at runtime from ntdll) are an EDR-evasion technique, not the default calling mechanism.

Resolving APIs via the PEB

There is no syscall table to index; shellcode locates functions by walking loaded modules in memory. The TEB/PEB live at fixed segment-relative offsets:

Arch Segment TEB PEB
x64 gs gs:[0x30] gs:[0x60]
x86 fs fs:[0x18] fs:[0x30]
  • fs/gs already point at the TEB base, so [0] is the TEB start. The TEB slot above is the self-pointer (NT_TIB.Self), the TEB's own flat address, for when you need it as a normal pointer. The PEB pointer sits at +0x60 (x64) / +0x30 (x86) from the TEB base.

  • Get the PEB: gs:[0x60] (x64) / fs:[0x30] (x86).

  • Follow PEB->Ldr->InMemoryOrderModuleList to find kernel32.dll's base.
  • Parse its PE export directory to resolve GetProcAddress / LoadLibraryA.
  • Use those to resolve and call everything else.

  • Functions are matched by hashing export names (avoids embedding ASCII strings and null bytes).

Resources

Syscall.sh