ELF Linking and Loading in Practice

Mon, Apr 22, 2024 tags: [ Programming Linux ]

I’ve programmed in C and C++, later in Rust and OCaml, for more than 12 years now. One of the most interesting aspects to me were the operating system, how code interacts with it, and how code is run. Like most people, I followed the typical journey: compile your first code using gcc -c hello.c, then link it using gcc -o hello hello.o, and finally run it using ./hello. With libsocket I published a library for both C and C++, facilitating using the sockets API on Unix systems - and as a library, I had to learn how to build a dynamic library (shared object, SO), and how to link against it. I’ve had my fair share of linker errors, too, but one thing stood out to me: linking and loading are complex processes, but generally so reliable that they almost always work. And if they don’t it’s usually my own fault - a missing translation unit, or something similar.

Recently I was faced with an issue at work that made me realize I don’t know enough about how linking works, both dynamic and static linking, what symbol tables are for; etc.: having to debug something that looked like clashing symbols in a shared library using Boost. I noticed there were not that many good resources to explain this at a level going beyond the basics, without having to go straight for the source code for ld.so.

In this article I wrote down what I learned about ELF linking and loading, in practical terms. That means: no source code of the tools needs to be examined; instead, everything can be found out through tools like readelf, nm, and objdump - maybe gdb for going deeper. Specifically I don’t want to try to explain how ELF looks or works beyond a level that is sufficient to understand linking and loading.

My approach is mostly a “stream of consciousness” - I’ve written down the different aspects of compiled code to visit, connecting them, sometimes repeating a snippet.

Necessary ELF knowledge

Sections: ELF files are divided into sections, like .text, .data, .bss, .rodata, .symtab, .strtab, and .shstrtab. The sections contain the actual program, like code, data, and symbols. Sections are described in the section header, which is an integral part of the ELF file.
Segments: Described by the program header table, segments are the parts of the ELF file that are loaded into memory. They contain the sections. Segments can be loadable (type LOAD), contain “dynamic linking information” (DYNAMIC), contain the interpreter required by a binary (INTERP), or be a note segment (NOTE), among others.
Symbol tables contain mappings from human-readable symbols to addresses in the file. For example, we care about where functions are located, especially during the linking process; later on this information is not necessary anymore and can be removed using a tool like strip. Symbol tables don’t contain the strings directly, but point into a string table. Typically, the symbol table is in a section called .symtab, and the string table in a section called .strtab.
The .dynamic section contains dynamic tags, which tell the dynamic linker ld.so (more on this later) on how to load the binary. For example, it contains the address of the DT_STRTAB and DT_SYMTAB entries, which point to the string table and symbol table, respectively, as well as a list of required shared objects.

Static linking

When you try to learn about static linking, you typically see an example like this:

// hello.c
void f(void);

int main() {
    f();
    return 0;
}

// f.c

#include <stdio.h>

void f(void) {
    puts("Hello, world!");
}

You compile these two files using gcc -c hello.c f.c, and then link them using gcc -o hello hello.o f.o.

Most articles will tell you: the compiler leaves some hole in the hello.o file for where the f() function call happens, and then the linker fixes it up. Sure enough - that works as a mental model. We know that gcc doesn’t know yet where f’s code lives in memory, so it leaves a placeholder.

But what happens within those object files? How does the linker know where to patch up the call to f(), that is, fill in the place holder?

As promised by some other articles, a linker - like ld or gold or mold - is really not such a complex piece of software. Lets examine the object file and find out where we can find the information the linker needs.

$ objdump --disassemble=main hello.o


hello.o:     file format elf64-x86-64


Disassembly of section .text:

0000000000000000 <main>:
   0:	55                   	push   %rbp
   1:	48 89 e5             	mov    %rsp,%rbp
   4:	e8 00 00 00 00       	call   9 <main+0x9>
   9:	b8 00 00 00 00       	mov    $0x0,%eax
   e:	5d                   	pop    %rbp
   f:	c3                   	ret

We see at offset 0x4 a call 9 instruction. This is the call to f(). The 9 is the relative offset to the next instruction, which is main+0x9. This doesn’t make sense! We don’t want to call right into the middle of main()!

How does it look in the final executable?

$ objdump --disassemble=main hello

hello:     file format elf64-x86-64


Disassembly of section .init:

Disassembly of section .plt:

Disassembly of section .text:

0000000000401126 <main>:
  401126:	55                   	push   %rbp
  401127:	48 89 e5             	mov    %rsp,%rbp
  40112a:	e8 07 00 00 00       	call   401136 <f>
  40112f:	b8 00 00 00 00       	mov    $0x0,%eax
  401134:	5d                   	pop    %rbp
  401135:	c3                   	ret

Disassembly of section .fini:

How did the address get here? The linker must have patched it up. But how did it know where f() is?

$ objdump -r hello.o


/home/lbo/test/hello.o:     file format elf64-x86-64

RELOCATION RECORDS FOR [.text]:
OFFSET           TYPE              VALUE
0000000000000005 R_X86_64_PLT32    f-0x0000000000000004

What this says is: At offset 0x5 within section .text, put the address of f minus 0x4. (Although to be honest I don’t understand the reason for the addend -4 here.) The linker reads the relocation records and simply pastes the function addresses into the placeholder sites.

See below for what the dynamic linker does - it’s quite similar, but the placeholder is not in the code, but a special table in memory.

Code in Memory

Okay - now we call a function f at address 0x401136. This is the address of the f() function in the final binary. Let’s verify it by checking the symbol table using nm; the symbol table is a section containing the names of all functions and global variables, which is useful for debugging and linking.

$ nm hello | grep ' f$'
0000000000401136 T f

The T means that f is a symbol referring to a function in the text section; a region within the ELF file containing executable code. The address 0x401136 is the address of the code of f. But what does 0x401136 mean? It’s not the address in the file, it’s the address in memory; because that’s where the CPU will jump when running the code. How does the linker know that f is at 0x401136?

Let’s check the section headers of the final hello executable (header added by me):

$ objdump -h hello | grep .text
 Idx Name         Size      VMA               LMA               File off  Algn
 13 .text         00000107  0000000000401040  0000000000401040  00001040  2**4

Not sure what VMA means - Virtual Memory Address? - but if .text starts at 0x401040, then f must be at 0x401136, i.e. 246 bytes into the .text section. main() is apparently at 230 bytes into that section.

$ objdump --disassemble -j .text hello

hello:     file format elf64-x86-64


Disassembly of section .text:

0000000000401040 <_start>:
  401040:	f3 0f 1e fa          	endbr64
  401044:	31 ed                	xor    %ebp,%ebp
  401046:	49 89 d1             	mov    %rdx,%r9
  401049:	5e                   	pop    %rsi
  40104a:	48 89 e2             	mov    %rsp,%rdx
  40104d:	48 83 e4 f0          	and    $0xfffffffffffffff0,%rsp
  401051:	50                   	push   %rax
  401052:	54                   	push   %rsp
  401053:	45 31 c0             	xor    %r8d,%r8d
  401056:	31 c9                	xor    %ecx,%ecx
  401058:	48 c7 c7 26 11 40 00 	mov    $0x401126,%rdi
  40105f:	ff 15 73 2f 00 00    	call   *0x2f73(%rip)        # 403fd8 <__libc_start_main@GLIBC_2.34>
  401065:	f4                   	hlt
  401066:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  40106d:	00 00 00 

< ... snip ... >

0000000000401126 <main>:
  401126:	55                   	push   %rbp
  401127:	48 89 e5             	mov    %rsp,%rbp
  40112a:	e8 07 00 00 00       	call   401136 <f>
  40112f:	b8 00 00 00 00       	mov    $0x0,%eax
  401134:	5d                   	pop    %rbp
  401135:	c3                   	ret

0000000000401136 <f>:
  401136:	55                   	push   %rbp
  401137:	48 89 e5             	mov    %rsp,%rbp
  40113a:	bf 10 20 40 00       	mov    $0x402010,%edi
  40113f:	e8 ec fe ff ff       	call   401030 <puts@plt>
  401144:	90                   	nop
  401145:	5d                   	pop    %rbp
  401146:	c3                   	ret

There it is - if we believe objdump that it’s just disassembling some raw bytes in the .text section, then f is located where it is because that section is just all the functions pasted right after each other, and f happens to be located last in that blob - at the given address.

Can we verify if that’s also where f is in memory?

For this the most accessible way is the maps file in procfs. We can see the memory layout of the hello process by running it in gdb, suspending it - so that our extremely short-lvied process is frozen - and the look at the file /proc/<pid>/maps:

$ cat /proc/242365/maps 
00400000-00401000 r--p 00000000 fd:03 271558987                          /home/lbo/test/hello
00401000-00402000 r-xp 00001000 fd:03 271558987                          /home/lbo/test/hello
00402000-00403000 r--p 00002000 fd:03 271558987                          /home/lbo/test/hello
00403000-00404000 r--p 00002000 fd:03 271558987                          /home/lbo/test/hello
00404000-00405000 rw-p 00003000 fd:03 271558987                          /home/lbo/test/hello
7ffff7dc0000-7ffff7dc2000 rw-p 00000000 00:00 0 
7ffff7dc2000-7ffff7de8000 r--p 00000000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7de8000-7ffff7f48000 r-xp 00026000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7f48000-7ffff7f96000 r--p 00186000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7f96000-7ffff7f9a000 r--p 001d3000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7f9a000-7ffff7f9c000 rw-p 001d7000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7f9c000-7ffff7fa6000 rw-p 00000000 00:00 0 
7ffff7fc3000-7ffff7fc7000 r--p 00000000 00:00 0                          [vvar]
7ffff7fc7000-7ffff7fc9000 r-xp 00000000 00:00 0                          [vdso]
7ffff7fc9000-7ffff7fca000 r--p 00000000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffff7fca000-7ffff7ff1000 r-xp 00001000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffff7ff1000-7ffff7ffb000 r--p 00028000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffff7ffb000-7ffff7ffd000 r--p 00031000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffff7ffd000-7ffff7fff000 rw-p 00033000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffffffdd000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

Loading

Okay - but how did those ELF bytes get to that memory offset? For this, a basic knowledge of virtual memory is required. Virtual memory just means that every process more or less thinks it has the entire memory address space to itself, so that it doesn’t matter that the code is always loaded at 0x401000 - every process has its own 0x401000. The CPU’s MMU does the job of isolating processes and translating addresses to physical memory space.

Let’s just look at the first couple lines: and there is our friend, 0x401000, with a size of 0x1000 bytes. This shows that the linker did its job, as expected, and f() can be called.

We can also correlate this with information contained in the ELF file and interpreted by the operating system. Let’s look at the segments and sections - I’ve filtered out the uninteresting parts:

# Sections
$ objdump -h ./hello


/home/lbo/test/hello:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
> 0 .interp       0000001c  0000000000400318  0000000000400318  00000318  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
...
  5 .dynsym       00000060  00000000004003e0  00000000004003e0  000003e0  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .dynstr       00000048  0000000000400440  0000000000400440  00000440  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  9 .rela.dyn     00000030  00000000004004c0  00000000004004c0  000004c0  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 10 .rela.plt     00000018  00000000004004f0  00000000004004f0  000004f0  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
>11 .init         0000001b  0000000000401000  0000000000401000  00001000  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 12 .plt          00000020  0000000000401020  0000000000401020  00001020  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 13 .text         00000107  0000000000401040  0000000000401040  00001040  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 14 .fini         0000000d  0000000000401148  0000000000401148  00001148  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
>15 .rodata       00000017  0000000000402000  0000000000402000  00002000  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
...
 20 .dynamic      000001d0  0000000000403e08  0000000000403e08  00002e08  2**3
                  CONTENTS, ALLOC, LOAD, DATA
 21 .got          00000010  0000000000403fd8  0000000000403fd8  00002fd8  2**3
                  CONTENTS, ALLOC, LOAD, DATA
 22 .got.plt      00000020  0000000000403fe8  0000000000403fe8  00002fe8  2**3
                  CONTENTS, ALLOC, LOAD, DATA
>23 .data         00000004  0000000000404008  0000000000404008  00003008  2**0
                  CONTENTS, ALLOC, LOAD, DATA
 24 .bss          00000004  000000000040400c  000000000040400c  0000300c  2**0
                  ALLOC
...

Each line marked with a > is a section with a start address directly corresponding to a mapping in the previous listing! Also consider the access rights: for example, .text is r-x, meaning it’s readable and executable, but not writable. That’s an important security feature.

Let’s check the segments as well - here we see the information used directly for mapping, including details like alignment and access rights (rwx etc.):

# segments
$ objdump -h ./hello


/home/lbo/test/hello:     file format elf64-x86-64

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000400040 paddr 0x0000000000400040 align 2**3
         filesz 0x00000000000002d8 memsz 0x00000000000002d8 flags r--
  INTERP off    0x0000000000000318 vaddr 0x0000000000400318 paddr 0x0000000000400318 align 2**0
         filesz 0x000000000000001c memsz 0x000000000000001c flags r--
>   LOAD off    0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**12
         filesz 0x0000000000000508 memsz 0x0000000000000508 flags r--
>   LOAD off    0x0000000000001000 vaddr 0x0000000000401000 paddr 0x0000000000401000 align 2**12
         filesz 0x0000000000000155 memsz 0x0000000000000155 flags r-x
>   LOAD off    0x0000000000002000 vaddr 0x0000000000402000 paddr 0x0000000000402000 align 2**12
         filesz 0x00000000000000fc memsz 0x00000000000000fc flags r--
    LOAD off    0x0000000000002df8 vaddr 0x0000000000403df8 paddr 0x0000000000403df8 align 2**12
         filesz 0x0000000000000214 memsz 0x0000000000000218 flags rw-
 DYNAMIC off    0x0000000000002e08 vaddr 0x0000000000403e08 paddr 0x0000000000403e08 align 2**3
         filesz 0x00000000000001d0 memsz 0x00000000000001d0 flags rw-
    NOTE off    0x0000000000000338 vaddr 0x0000000000400338 paddr 0x0000000000400338 align 2**3
         filesz 0x0000000000000040 memsz 0x0000000000000040 flags r--
    NOTE off    0x0000000000000378 vaddr 0x0000000000400378 paddr 0x0000000000400378 align 2**2
         filesz 0x0000000000000044 memsz 0x0000000000000044 flags r--
0x6474e553 off    0x0000000000000338 vaddr 0x0000000000400338 paddr 0x0000000000400338 align 2**3
         filesz 0x0000000000000040 memsz 0x0000000000000040 flags r--
EH_FRAME off    0x0000000000002018 vaddr 0x0000000000402018 paddr 0x0000000000402018 align 2**2
         filesz 0x0000000000000034 memsz 0x0000000000000034 flags r--
   STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
   RELRO off    0x0000000000002df8 vaddr 0x0000000000403df8 paddr 0x0000000000403df8 align 2**0
         filesz 0x0000000000000208 memsz 0x0000000000000208 flags r--

Similarly we find entries for addresses 0x400000, 0x401000, and 0x402000. Segments for .got and .got.plt also show up, but not at clean offsets.

Loading dynamic objects

Let’s accept that the operating system maps the ELF into memory. But how does the code from dynamic libraries get to where it is needed? My all-time favorite tool for investigating questions like this is strace, which shows syscalls as they happen. Here the only library is libc.so.6 - that’s a risky choice, because it’s a very large and complex library; we should assume that all tricks in the book are used to optimize it, and it might not look like any normal shared object.

$ strace ./hello
strace ./hello 
execve("./hello", ["./hello"], 0x7ffcf80b3880 /* 59 vars */) = 0
brk(NULL)                               = 0x15c6000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffede418830) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=115979, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 115979, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f37aabfb000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \203\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \203\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2369656, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f37aabf9000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1973104, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f37aaa17000
mmap(0x7f37aaa3d000, 1441792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f37aaa3d000
mmap(0x7f37aab9d000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x186000) = 0x7f37aab9d000
mmap(0x7f37aabeb000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d3000) = 0x7f37aabeb000
mmap(0x7f37aabf1000, 31600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f37aabf1000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f37aaa15000
arch_prctl(ARCH_SET_FS, 0x7f37aabfa680) = 0
set_tid_address(0x7f37aabfa950)         = 242543
set_robust_list(0x7f37aabfa960, 24)     = 0
rseq(0x7f37aabfafa0, 0x20, 0, 0x53053053) = 0
mprotect(0x7f37aabeb000, 16384, PROT_READ) = 0
mprotect(0x403000, 4096, PROT_READ)     = 0
mprotect(0x7f37aac4a000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7f37aabfb000, 115979)          = 0
newfstatat(1, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x5), ...}, AT_EMPTY_PATH) = 0
getrandom("\x4b\x7b\xee\xee\x17\xc6\x15\x0e", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x15c6000
brk(0x15e7000)                          = 0x15e7000
write(1, "Hello\n", 6Hello
)                  = 6
write(1, "\n", 1
)                       = 1
exit_group(0)                           = ?
+++ exited with 0 +++

What is mmap()? Its manpage shows the following function signature:

void *mmap(void addr[.length], size_t length, int prot, int flags, int fd, off_t offset);

where

addr is a “hint” to the operating system where to create the mapping;
length is the length of the mapping;
prot is the protection flags, like PROT_READ, PROT_WRITE, PROT_EXEC;
flags are the mapping flags, like MAP_PRIVATE, MAP_ANONYMOUS, MAP_FIXED;
fd is the file descriptor of the file to map; and
offset is the offset into the file to start mapping.

mmap() is a syscall asking the operating system to either provide a new memory region (which can be used for whatever we want), or to map a file into memory. A file mapped into memory can be accessed just like normal memory, but the operating system will ensure that we read from the file when we access the memory, and write to the file when we write to it (depending on the flags). In this context, we are only interested in reads. Cleverly, a file being mapped to memory doesn’t mean that the file is loaded in memory - it may be loaded on-demand, for example, and removed again from memory when it’s not needed anymore and memory is becoming scarce.

We don’t need to understand all of this to understand what happens, but let’s compare the addresses from strace with the /proc/<pid>/maps file - they don’t match! It turns out that libraries are loaded at random addresses, as a security measure. But we can make our life easier and run the following - after all, /bin/cat is also a dynamically linked binary, and we just trace cat as it outputs its own mappings:

$ strace cat /proc/self/maps
execve("/usr/bin/cat", ["cat", "/proc/self/maps"], 0x7ffe459ab028 /* 59 vars */) = 0
brk(NULL)                               = 0x561628201000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffef9889430) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=115979, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 115979, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f471c482000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \203\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2369656, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f471c480000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1973104, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f471c29e000
mmap(0x7f471c2c4000, 1441792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f471c2c4000
mmap(0x7f471c424000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x186000) = 0x7f471c424000
mmap(0x7f471c472000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d3000) = 0x7f471c472000
mmap(0x7f471c478000, 31600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f471c478000
close(3)                                = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f471c29b000

< ... locales being set up etc. ...>

mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f471c279000
read(3, "561626b15000-561626b17000 r--p 0"..., 131072) = 3387
write(1, "561626b15000-561626b17000 r--p 0"..., 3387561626b15000-561626b17000 r--p 00000000 fd:01 134505691                  /usr/bin/cat

< ... this is the content of /proc/self/maps: ... >

561626b17000-561626b1c000 r-xp 00002000 fd:01 134505691                  /usr/bin/cat
561626b1c000-561626b1e000 r--p 00007000 fd:01 134505691                  /usr/bin/cat
561626b1e000-561626b1f000 r--p 00008000 fd:01 134505691                  /usr/bin/cat
561626b1f000-561626b20000 rw-p 00009000 fd:01 134505691                  /usr/bin/cat
561628201000-561628222000 rw-p 00000000 00:00 0                          [heap]
7f471bfa8000-7f471c000000 r--p 00000000 fd:01 202251554                  /usr/lib/locale/C.utf8/LC_CTYPE
7f471c000000-7f471c278000 r--p 00000000 fd:01 243239                     /usr/lib/locale/en_US.utf8/LC_COLLATE
7f471c279000-7f471c29e000 rw-p 00000000 00:00 0 
7f471c29e000-7f471c2c4000 r--p 00000000 fd:01 134859782                  /usr/lib64/libc.so.6
7f471c2c4000-7f471c424000 r-xp 00026000 fd:01 134859782                  /usr/lib64/libc.so.6
7f471c424000-7f471c472000 r--p 00186000 fd:01 134859782                  /usr/lib64/libc.so.6
7f471c472000-7f471c476000 r--p 001d3000 fd:01 134859782                  /usr/lib64/libc.so.6
7f471c476000-7f471c478000 rw-p 001d7000 fd:01 134859782                  /usr/lib64/libc.so.6
7f471c478000-7f471c482000 rw-p 00000000 00:00 0 
7f471c48e000-7f471c48f000 r--p 00000000 fd:01 243243                     /usr/lib/locale/en_GB.utf8/LC_NUMERIC
7f471c48f000-7f471c490000 r--p 00000000 fd:01 134859770                  /usr/lib/locale/en_GB.utf8/LC_TIME
7f471c490000-7f471c491000 r--p 00000000 fd:01 134785226                  /usr/lib/locale/en_GB.utf8/LC_MONETARY
7f471c491000-7f471c492000 r--p 00000000 fd:01 243241                     /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
7f471c492000-7f471c493000 r--p 00000000 fd:01 243244                     /usr/lib/locale/en_GB.utf8/LC_PAPER
7f471c493000-7f471c494000 r--p 00000000 fd:01 243242                     /usr/lib/locale/en_US.utf8/LC_NAME
7f471c494000-7f471c495000 r--p 00000000 fd:01 134784710                  /usr/lib/locale/en_GB.utf8/LC_ADDRESS
7f471c495000-7f471c496000 r--p 00000000 fd:01 134785285                  /usr/lib/locale/en_GB.utf8/LC_TELEPHONE
7f471c496000-7f471c497000 r--p 00000000 fd:01 243240                     /usr/lib/locale/en_GB.utf8/LC_MEASUREMENT
7f471c497000-7f471c49e000 r--s 00000000 fd:01 18923                      /usr/lib64/gconv/gconv-modules.cache
7f471c49e000-7f471c49f000 r--p 00000000 fd:01 243224                     /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
7f471c49f000-7f471c4a0000 r--p 00000000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7f471c4a0000-7f471c4c7000 r-xp 00001000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7f471c4c7000-7f471c4d1000 r--p 00028000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7f471c4d1000-7f471c4d3000 r--p 00031000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7f471c4d3000-7f471c4d5000 rw-p 00033000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffef986a000-7ffef988b000 rw-p 00000000 00:00 0                          [stack]
7ffef999c000-7ffef99a0000 r--p 00000000 00:00 0                          [vvar]
7ffef99a0000-7ffef99a2000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
) = 3387
read(3, "", 131072)                     = 0
munmap(0x7f471c279000, 139264)          = 0
close(3)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

And now we see: /lib64/lib.so.6, the C library, is opened with file descriptor 3. Some header information appears to be read first, and then the second mmap() asks to map the first 1973104 bytes of the file into memory starting at offset 0. The OS returns the address of tha mapping: 0x7f471c29e000. We find this address again in the maps file shown further down; here it is a read-only mapping of a part of the C library, specifically almost the whole file starting at offset 0.

Further down, more mmap()s are issued, with the flag MAP_FIXED set. This tells the OS to remove any existing mapping in that region and replace it with the new mapping. For example, there’s an mmap() specifying offset 0x26000. This, according to readelf, is the offset of the .plt segment (which I will get back to later).

Let’s take a look at the program header table of the C library, which defines the segments of the file. Each segment can contain multiple sections, which are mapped into memory as a whole.

Let’s again check the program header and the sections, this time of the C library - not of the program we ran:

# Dump program headers
$ objdump -p /usr/lib64/libc.so.6


/usr/lib64/libc.so.6:     file format elf64-x86-64

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000000040 paddr 0x0000000000000040 align 2**3
         filesz 0x0000000000000310 memsz 0x0000000000000310 flags r--
  INTERP off    0x00000000001abe20 vaddr 0x00000000001abe20 paddr 0x00000000001abe20 align 2**4
         filesz 0x000000000000001c memsz 0x000000000000001c flags r--
    LOAD off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**12
         filesz 0x0000000000025ac0 memsz 0x0000000000025ac0 flags r--
    LOAD off    0x0000000000026000 vaddr 0x0000000000026000 paddr 0x0000000000026000 align 2**12
         filesz 0x000000000015facd memsz 0x000000000015facd flags r-x
    LOAD off    0x0000000000186000 vaddr 0x0000000000186000 paddr 0x0000000000186000 align 2**12
         filesz 0x000000000004d4b7 memsz 0x000000000004d4b7 flags r--
    LOAD off    0x00000000001d3ca0 vaddr 0x00000000001d4ca0 paddr 0x00000000001d4ca0 align 2**12
         filesz 0x0000000000004a28 memsz 0x000000000000ced0 flags rw-
 DYNAMIC off    0x00000000001d6980 vaddr 0x00000000001d7980 paddr 0x00000000001d7980 align 2**3
         filesz 0x0000000000000210 memsz 0x0000000000000210 flags rw-
    NOTE off    0x0000000000000350 vaddr 0x0000000000000350 paddr 0x0000000000000350 align 2**3
         filesz 0x0000000000000050 memsz 0x0000000000000050 flags r--
    NOTE off    0x00000000000003a0 vaddr 0x00000000000003a0 paddr 0x00000000000003a0 align 2**2
         filesz 0x0000000000000044 memsz 0x0000000000000044 flags r--
     TLS off    0x00000000001d3ca0 vaddr 0x00000000001d4ca0 paddr 0x00000000001d4ca0 align 2**3
         filesz 0x0000000000000010 memsz 0x0000000000000088 flags r--
0x6474e553 off    0x0000000000000350 vaddr 0x0000000000000350 paddr 0x0000000000000350 align 2**3
         filesz 0x0000000000000050 memsz 0x0000000000000050 flags r--
EH_FRAME off    0x00000000001abe3c vaddr 0x00000000001abe3c paddr 0x00000000001abe3c align 2**2
         filesz 0x000000000000763c memsz 0x000000000000763c flags r--
   STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
   RELRO off    0x00000000001d3ca0 vaddr 0x00000000001d4ca0 paddr 0x00000000001d4ca0 align 2**0
         filesz 0x0000000000003360 memsz 0x0000000000003360 flags r--

The LOAD segment at offset 0x26000 is recognizable from the fourth observed mmap(), and so is the LOAD segment at offset 0x186000 (that was the fifth mmap()). There are also some mmap()s with unknown addresses, presumably indexing into the middle of some segments. The access rights shown here also match the mmap() flags observed in strace.

Can we learn more from the sections?

$ objdump -h /usr/lib64/libc.so.6


/usr/lib64/libc.so.6:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .note.gnu.property 00000050  0000000000000350  0000000000000350  00000350  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .note.gnu.build-id 00000024  00000000000003a0  00000000000003a0  000003a0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .note.ABI-tag 00000020  00000000000003c4  00000000000003c4  000003c4  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .hash         00004004  00000000000003e8  00000000000003e8  000003e8  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .gnu.hash     000047d8  00000000000043f0  00000000000043f0  000043f0  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .dynsym       00012090  0000000000008bc8  0000000000008bc8  00008bc8  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .dynstr       000081b3  000000000001ac58  000000000001ac58  0001ac58  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .gnu.version  0000180c  0000000000022e0c  0000000000022e0c  00022e0c  2**1
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  8 .gnu.version_d 00000588  0000000000024618  0000000000024618  00024618  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  9 .gnu.version_r 00000040  0000000000024ba0  0000000000024ba0  00024ba0  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 10 .rela.dyn     00000810  0000000000024be0  0000000000024be0  00024be0  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 11 .rela.plt     000005d0  00000000000253f0  00000000000253f0  000253f0  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 12 .relr.dyn     00000100  00000000000259c0  00000000000259c0  000259c0  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 13 .plt          000003f0  0000000000026000  0000000000026000  00026000  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 14 .plt.sec      000003e0  00000000000263f0  00000000000263f0  000263f0  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 15 .text         0015f2cd  0000000000026800  0000000000026800  00026800  2**6
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 16 .rodata       00025da4  0000000000186000  0000000000186000  00186000  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
< ... and so on ... >

Here we find the .plt section at offset 0x26000 and the .rodata section at offset 0x186000. The PLT (Procedure Linkage Table) is a table of pointers to functions in shared libraries, used to call functions in shared libraries, and I will get back to this later. Also note that several of the sections in this table are located directly adjacent to each other, and the file header table tells us that several of them are loaded into memory as a single segment.

I wouldn’t go so far to say that all of this is trivial or even simple, but it does boil down to some properly aligned memory mappings backed by the executable and dynamic libraries we assembled before. The component taking care of the memory mappings is the dynamic loader ld.so, which is invoked by the kernel.

But wait… how does the kernel know where to find ld.so? This is where the .interp section comes in. The .interp section is a simple pointer into the ELF file. In the second to last listing, for hello, the file off value of the .interp section is 0x318. The .interp section coincides with the segment marked INTERP, as well.

/home/lbo/test/hello:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .interp       0000001c  0000000000400318  0000000000400318  00000318  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA

Let’s check:

$ xxd -s 0x318 -l 64 ./hello
00000318: 2f6c 6962 3634 2f6c 642d 6c69 6e75 782d  /lib64/ld-linux-
00000328: 7838 362d 3634 2e73 6f2e 3200 0000 0000  x86-64.so.2.....
00000338: 0400 0000 3000 0000 0500 0000 474e 5500  ....0.......GNU.
00000348: 0280 00c0 0400 0000 0100 0000 0000 0000  ................

Wonderful!

And how does ld.so know where to find the shared libraries? The elf(5) manpage has some information:

       .dynamic
              This section holds dynamic linking information.  The section's attributes will include the SHF_ALLOC bit.  Whether the SHF_WRITE  bit  is  set  is
              processor-specific.  This section is of type SHT_DYNAMIC.  See the attributes above.

       .dynstr
              This  section  holds  strings needed for dynamic linking, most commonly the strings that represent the names associated with symbol table entries.
              This section is of type SHT_STRTAB.  The attribute type used is SHF_ALLOC.

       .dynsym
              This section holds the dynamic linking symbol table.  This section is of type SHT_DYNSYM.  The attribute used is SHF_ALLOC.

It’s not overly detailed, but we can cobble something together from this, as well as the description of Symbol Tables:

String and symbol tables
       String table sections hold null-terminated character sequences, commonly called strings.  The object file uses these strings to represent symbol and sec‐
       tion names.  One references a string as an index into the string table section.  The first byte, which is index zero, is defined  to  hold  a  null  byte
       ('\0').  Similarly, a string table's last byte is defined to hold a null byte, ensuring null termination for all strings.

       An object file's symbol table holds information needed to locate and relocate a program's symbolic definitions and references.  A symbol table index is a
       subscript into this array.

           typedef struct {
               uint32_t      st_name;
               unsigned char st_info;
               unsigned char st_other;
               uint16_t      st_shndx;
               Elf64_Addr    st_value;
               uint64_t      st_size;
           } Elf64_Sym;

We can inspect the dynamic section like this, and find information on the shared library dependencies, as well as information used by the dynamic linker for setting up the program’s address space.

$ readelf -d ./hello

Dynamic section at offset 0x2e08 contains 24 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x401000
 0x000000000000000d (FINI)               0x401148
 0x0000000000000019 (INIT_ARRAY)         0x403df8
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x403e00
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x4003c0
 0x0000000000000005 (STRTAB)             0x400440
 0x0000000000000006 (SYMTAB)             0x4003e0
 0x000000000000000a (STRSZ)              72 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x403fe8
 0x0000000000000002 (PLTRELSZ)           24 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x4004f0
 0x0000000000000007 (RELA)               0x4004c0
 0x0000000000000008 (RELASZ)             48 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x400490
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x400488
 0x0000000000000000 (NULL)               0x0

At the top you see the entry for the C library dependency. The other entries specify the addresses of the dynamic Linking information, like the symbol table, the string table, the hash table, and the PLT, when loaded in memory. For example, if you search for 0x403fe8 further down, this address shows up as the start of the .got.plt section loaded in memory.

The hello binary is a bit boring, but a test binary from the uvco project has a few more entries here:

Dynamic section at offset 0x59d28 contains 37 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libboost_program_options.so.1.81.0]
 0x0000000000000001 (NEEDED)             Shared library: [libboost_log_setup.so.1.81.0]
 0x0000000000000001 (NEEDED)             Shared library: [libuv.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libfmt.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libboost_log.so.1.81.0]
 0x0000000000000001 (NEEDED)             Shared library: [libboost_chrono.so.1.81.0]
 0x0000000000000001 (NEEDED)             Shared library: [libboost_filesystem.so.1.81.0]
 0x0000000000000001 (NEEDED)             Shared library: [libboost_atomic.so.1.81.0]
 0x0000000000000001 (NEEDED)             Shared library: [libboost_regex.so.1.81.0]
 0x0000000000000001 (NEEDED)             Shared library: [libboost_thread.so.1.81.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
< ... and so on ...>

As promised, I don’t want to go further into the details of ELF files, but we get the idea: there are some sections - byte ranges in the ELF file - containing tables describing which symbols are in the file, and which libraries are needed to run the program.

More on Symbol Tables

A nice tool for inspecting symbol tables is nm. Let’s look at the symbol table of hello:

$ nm hello
000000000040039c r __abi_tag
000000000040400c B __bss_start
000000000040400c b completed.0
0000000000404008 D __data_start
0000000000404008 W data_start
0000000000401080 t deregister_tm_clones
0000000000401070 T _dl_relocate_static_pie
00000000004010f0 t __do_global_dtors_aux
0000000000403e00 d __do_global_dtors_aux_fini_array_entry
0000000000402008 R __dso_handle
0000000000403e08 d _DYNAMIC
000000000040400c D _edata
0000000000404010 B _end
0000000000401136 T f
0000000000401148 T _fini
0000000000401120 t frame_dummy
0000000000403df8 d __frame_dummy_init_array_entry
00000000004020f8 r __FRAME_END__
0000000000403fe8 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
0000000000402018 r __GNU_EH_FRAME_HDR
0000000000401000 T _init
0000000000402000 R _IO_stdin_used
                 U __libc_start_main@GLIBC_2.34
0000000000401126 T main
                 U puts@GLIBC_2.2.5
00000000004010b0 t register_tm_clones
0000000000401040 T _start
0000000000404010 D __TMC_END__

There’s a lot going on, but we do find our main, f, and puts functions. t/T means that the symbol points to a function in the .text section, whereas U notifies us that the symbol is undefined. The entire set of symbol types is described in the nm(1) manual page. Lowercase symbols have local visibility, whereas uppercase symbols are global. More on symbol visibility can be found in elf(5), field st_other of symbol table entries. In short, global symbols are visible across ELF files, whereas local symbols are only visible within the file they are defined in. This is important for shared libraries, for example, where you might not want to override weak symbols in binaries using the library.

There is also versioning information, like GLIBC_2.34, which tells us that the symbol is part of the GNU C library version 2.34. Weak symbols (w/W) are also interesting - they are defined in the same file, but can be overridden by symbols in dynamic libraries during loading time. This allows us to define “hooks”, for example to override a default function to change some behavior.

But what’s a symbol table? This seems simple, but confused me for quite some time.

As we saw in the elf(5) manpage, a symbol table is an array of Elf64_Sym structures. Each structure contains a name index, the address of the associated symbol, and some metadata like its size, type, and binding. The name index points to a string tale, which is a section in the ELF file containing null-terminated strings.

In our hello file, the .symtab section contains the symbol table entries, which point to strings in the .strtab section. These sections are removed using the strip utility, if you want to shrink your binary’s size. objdump can show us the symbol table of an ELF file in even greater detail, for example which sections symbols are located in:

$ objdump -t hello

hello:     file format elf64-x86-64

SYMBOL TABLE:
0000000000000000 l    df *ABS*	0000000000000000              crt1.o
000000000040039c l     O .note.ABI-tag	0000000000000020              __abi_tag
0000000000000000 l    df *ABS*	0000000000000000              crtbegin.o
0000000000401080 l     F .text	0000000000000000              deregister_tm_clones
00000000004010b0 l     F .text	0000000000000000              register_tm_clones
00000000004010f0 l     F .text	0000000000000000              __do_global_dtors_aux
000000000040400c l     O .bss	0000000000000001              completed.0
0000000000403e00 l     O .fini_array	0000000000000000              __do_global_dtors_aux_fini_array_entry
0000000000401120 l     F .text	0000000000000000              frame_dummy
0000000000403df8 l     O .init_array	0000000000000000              __frame_dummy_init_array_entry
0000000000000000 l    df *ABS*	0000000000000000              hello.c
0000000000000000 l    df *ABS*	0000000000000000              f.c
0000000000000000 l    df *ABS*	0000000000000000              crtend.o
00000000004020f8 l     O .eh_frame	0000000000000000              __FRAME_END__
0000000000000000 l    df *ABS*	0000000000000000              
0000000000403e08 l     O .dynamic	0000000000000000              _DYNAMIC
0000000000402018 l       .eh_frame_hdr	0000000000000000              __GNU_EH_FRAME_HDR
0000000000403fe8 l     O .got.plt	0000000000000000              _GLOBAL_OFFSET_TABLE_
0000000000000000       F *UND*	0000000000000000              __libc_start_main@GLIBC_2.34
0000000000404008  w      .data	0000000000000000              data_start
0000000000000000       F *UND*	0000000000000000              puts@GLIBC_2.2.5
000000000040400c g       .data	0000000000000000              _edata
0000000000401148 g     F .fini	0000000000000000              .hidden _fini
0000000000401136 g     F .text	0000000000000011              f
0000000000404008 g       .data	0000000000000000              __data_start
0000000000000000  w      *UND*	0000000000000000              __gmon_start__
0000000000402008 g     O .rodata	0000000000000000              .hidden __dso_handle
0000000000402000 g     O .rodata	0000000000000004              _IO_stdin_used
0000000000404010 g       .bss	0000000000000000              _end
0000000000401070 g     F .text	0000000000000005              .hidden _dl_relocate_static_pie
0000000000401040 g     F .text	0000000000000026              _start
000000000040400c g       .bss	0000000000000000              __bss_start
0000000000401126 g     F .text	0000000000000010              main
0000000000404010 g     O .data	0000000000000000              .hidden __TMC_END__
0000000000401000 g     F .init	0000000000000000              .hidden _init

This is the same information as nm provides, but in a different format. The objdump output is more verbose, and I find it more useful.

Next we’ll find out how dynamic symbols like puts@GLIBC_2.2.5 are resolved during start-up.

Dynamic Linking

Dynamic linking is the process of linking a program at runtime, rather than at compile time. This allows us to use shared objects that - as shown earlier - are loaded into memory once, instead of being copied into every executable. We saw how ld.so maps shared objects into a process address space, but we don’t yet know how the program code knows where to call the functions in the shared objects.

So just like during the static linking process, the dynamic linker needs to resolve symbols. This is done in a fairly similar way as I’ve shown above for static linking, but with a few twists.

objdump can show us dynamic relocation information:

$ objdump -R hello


hello:     file format elf64-x86-64

DYNAMIC RELOCATION RECORDS
OFFSET           TYPE              VALUE
0000000000403fd8 R_X86_64_GLOB_DAT  __libc_start_main@GLIBC_2.34
0000000000403fe0 R_X86_64_GLOB_DAT  __gmon_start__@Base
0000000000404000 R_X86_64_JUMP_SLOT  puts@GLIBC_2.2.5

We recognize again - a symbol and its offset. So it’s reasonable to assume that the dynamic loader will read the dynamic relocation section, and write the symbols’ addresses to the specified offsets, just like during the static linking process.

Let’s check what’s at 0x404000, the offset the linker is supposed to treat with the resolved address: Is it the call site of puts? That would seem inefficient if we have more than one call…

$ objdump -j .text --disassemble=f hello

hello:     file format elf64-x86-64


Disassembly of section .text:

0000000000401136 <f>:
  401136:	55                   	push   %rbp
  401137:	48 89 e5             	mov    %rsp,%rbp
  40113a:	bf 10 20 40 00       	mov    $0x402010,%edi
  40113f:	e8 ec fe ff ff       	call   401030 <puts@plt>
  401144:	90                   	nop
  401145:	5d                   	pop    %rbp
  401146:	c3                   	ret

No, it isn’t? The call site already seems okay, it calls a symbol at 0x401030 named puts@plt. Mysterious! What’s there? According to the section headers (objdump -h) this points into the .plt section:

 12 .plt          00000020  0000000000401020  0000000000401020  00001020  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
<snip>
 21 .got          00000010  0000000000403fd8  0000000000403fd8  00002fd8  2**3
                  CONTENTS, ALLOC, LOAD, DATA
 22 .got.plt      00000020  0000000000403fe8  0000000000403fe8  00002fe8  2**3
                  CONTENTS, ALLOC, LOAD, DATA

The PLT

The .plt section contains code:

$ objdump -j .plt --disassemble hello

hello:     file format elf64-x86-64


Disassembly of section .plt:

0000000000401020 <puts@plt-0x10>:
  401020:	ff 35 ca 2f 00 00    	push   0x2fca(%rip)        # 403ff0 <_GLOBAL_OFFSET_TABLE_+0x8>
  401026:	ff 25 cc 2f 00 00    	jmp    *0x2fcc(%rip)        # 403ff8 <_GLOBAL_OFFSET_TABLE_+0x10>
  40102c:	0f 1f 40 00          	nopl   0x0(%rax)

0000000000401030 <puts@plt>:
  401030:	ff 25 ca 2f 00 00    	jmp    *0x2fca(%rip)        # 404000 <puts@GLIBC_2.2.5>
  401036:	68 00 00 00 00       	push   $0x0
  40103b:	e9 e0 ff ff ff       	jmp    401020 <_init+0x20>

The .got.plt is the Global Offset Table, which just appears to contain some little-endian pointers, which currently point back into the .plt section (36104000 is 0x401036). Also some null pointers:

$ objdump -j .got.plt -s hello

hello:     file format elf64-x86-64

Contents of section .got.plt:
 403fe8 083e4000 00000000 00000000 00000000  .>@.............
 403ff8 00000000 00000000 36104000 00000000  ........6.@.....

Remember, these sections are mapped as follows:

$ cat /proc/253143/maps
00400000-00401000 r--p 00000000 fd:03 268658691                          /home/lbo/test/hello
00401000-00402000 r-xp 00001000 fd:03 268658691                          /home/lbo/test/hello << .plt
00402000-00403000 r--p 00002000 fd:03 268658691                          /home/lbo/test/hello
00403000-00404000 r--p 00002000 fd:03 268658691                          /home/lbo/test/hello << .got.plt
00404000-00405000 rw-p 00003000 fd:03 268658691                          /home/lbo/test/hello << .got.plt
7ffff8dc0000-7ffff7dc2000 rw-p 00000000 00:00 0 
7ffff7dc2000-7ffff7de8000 r--p 00000000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7de8000-7ffff7f48000 r-xp 00026000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7f48000-7ffff7f96000 r--p 00186000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7f96000-7ffff7f9a000 r--p 001d3000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7f9a000-7ffff7f9c000 rw-p 001d7000 fd:01 134859782                  /usr/lib64/libc.so.6
7ffff7f9c000-7ffff7fa6000 rw-p 00000000 00:00 0 
7ffff7fc3000-7ffff7fc7000 r--p 00000000 00:00 0                          [vvar]
7ffff7fc7000-7ffff7fc9000 r-xp 00000000 00:00 0                          [vdso]
7ffff7fc9000-7ffff7fca000 r--p 00000000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffff7fca000-7ffff7ff1000 r-xp 00001000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffff7ff1000-7ffff7ffb000 r--p 00028000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffff7ffb000-7ffff7ffd000 r--p 00031000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffff7ffd000-7ffff7fff000 rw-p 00033000 fd:01 134299925                  /usr/lib64/ld-linux-x86-64.so.2
7ffffffdd000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

If we look at the .plt assembly, we see that the code for puts@plt jumps to the address stored for puts in the .got.plt section, which at the moment is set to 0x301036. Wait! That’s just the next instruction! This is a trick to make the first call continue in the puts@plt function, which then jumps to 0x401020, just above. Before doing so, it pushes 0x0 onto the stack, signaling that this is the first call to puts (if there were more functions used from glibc, the next stub would push 0x1, and so on). The very first entry of plt is a special case, and is used to resolve the address of the desired function by calling into the dynamic linker.

Because now things happen at run-time, I’ve started gdb to see what’s going on:

$ gdb ./hello
# Familiar memory contents - just like read by objdump:
(gdb) disas 0x401020 , 0x401040
Dump of assembler code from 0x401020 to 0x401040:
   0x0000000000401020:	push   0x2fca(%rip)        # 0x403ff0
   0x0000000000401026:	jmp    *0x2fcc(%rip)        # 0x403ff8
   0x000000000040102c:	nopl   0x0(%rax)
   0x0000000000401030 <puts@plt+0>:	jmp    *0x2fca(%rip)        # 0x404000 <puts@got.plt>
   0x0000000000401036 <puts@plt+6>:	push   $0x0
   0x000000000040103b <puts@plt+11>:	jmp    0x401020
End of assembler dump.

# What's in the .got.plt section?
(gdb) x/4gx 0x403ff0
0x403ff0:	0x0000000000000000	0x0000000000000000
0x404000 <puts@got.plt>:	0x0000000000401036	0x0000000000000000

# Apparently nothing... let's run the binary and stop in main()

(gdb) b main
Breakpoint 1 at 0x40112a: file hello.c, line 3.
(gdb) r
Starting program: /home/lbo/test/hello 
[Thread debugging using libthread_db enabled]                                                                                                                      
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, main () at hello.c:3
3	int main(void) { f(); return 0; }

# And try again:

(gdb) x/4gx 0x403ff0
0x403ff0:	0x00007ffff7ffe2c0	0x00007ffff7fdc230
0x404000 <puts@got.plt>:	0x0000000000401036	0x0000000000000000

# Interesting!

After running, the .got.plt section contains 1) pointers into the dynamic loader (the first entry, pointing to an RW region, the second into an an RX region), and 2) the address of the puts function. The dynamic loader will somehow resolve the address of puts and write it into the .got.plt section:

(gdb) b puts
Breakpoint 2 at 0x7ffff7e3d390: file ioputs.c, line 33.                                                                                                            
(gdb) c
Continuing.

Breakpoint 2, __GI__IO_puts (str=0x402010 "Hello\n") at ioputs.c:33
33	{
(gdb) x/4gx 0x403ff0
0x403ff0:	0x00007ffff7ffe2c0	0x00007ffff7fdc230
0x404000 <puts@got.plt>:	0x00007ffff7e3d390	0x0000000000000000

As we can see by entering puts() and then checking again the .got.plt section. Now the address of puts has been resolved and written into the .got.plt section; the 0x7ffff7e3d390 is the address of the puts function in the C library, as we can guess from the mapping file sown above; that address is in the RX region of the C library, which is the .text section.

If we go back to the .plt contents, we see that the jmp instruction at 0x401030 now jumps to the resolved address of puts, without further overhead:

# gdb disassemble 0x401020, 0x401040
<snip>
   0x0000000000401030 <puts@plt+0>:	jmp    *0x2fca(%rip)        # 0x404000 <puts@got.plt>
<snap>

One thing I don’t understand: according to maps, the addresses up to 0x404000 are read-only; but obviously they are modified by the dynamic loader, as seen in gdb. Mystery!

Conclusion for now

This has been quite a journey, and we’ve still merely scratched the surface. My lightbulb moment was fairly benign, when I checked the mmap addresses and offsets with the maps file and the ELF file sections. Maybe I’ll revisit this topic later and extend or amend the information here.