As an engineer on the Greenplum Release Engineering team, I recently had the opportunity to delve into Postgres’s build system. Greenplum Server is based on Postgres and inherits the build system upstream. The GP-Releng team is creating a relocatable version of Greenplum Server, which led us to explore how to do a relocatable version of Postgres. Relocatable (Relocated) mentioned in this paper refers to in the case of without recompiling install Postgres, after changing the installation path, the program can still run normally, and do not need to set LD_LIBRARY_PATH can cause the program to find the correct Shared library. Consider the following what-if scenarios to highlight key ideas and discovery processes, rather than obsessing over the details of our team’s use cases.

11.8 build Postgres

Cardenia is a graduate student at an American university laboratory. She wanted to use Postgres for data analysis, but she didn’t have root access to the lab’s shared server. Not wanting to disturb the server’s silver-haired administrator, she decided to build, install, and run Postgres in her home directory. Since the data analysis uses Perl modules, Postgres needs to be configured to use PL/Perl.

Mkdir -p. ~ / local/SRC CD ~ / local/SRC curl https://ftp.postgresql.org/pub/source/v11.8/postgresql-11.8.tar.bz2 | \ tar - xj CD. / postgresql 11.8. / configure \ -- prefix = / home/cardenia/local/postgres \ - with - perl make -j make installCopy the code

Postgres has now been successfully built and installed. She decided to do a quick test

$ cd ~ $ export PATH="${HOME}/.local/postgres/bin:${PATH}" $ initdb ~/test ... $ pg_ctl -D ~/test -l logfile start waiting for server to start.... done server started $ createdb $ psql -c 'SELECT version(); ' version --------------------------------------------------------------------------------------------------------- PostgreSQL 11.8 on x86_64-PC-linux-GNU, compiled by GCC (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit (1 row)Copy the code

It looks like the Postgres server is up and running, but she also wants to make sure PL/Perl is working as well. Start PSQL and create PL/Perl functions

-- taken from https://www.postgresql.org/docs/11/plperl-funcs.html
CREATE EXTENSION plperl;
CREATE FUNCTION perl_max (integer, integer) RETURNS integer AS $$
    if ($_[0] > $_[1]) { return $_[0]; }
    return $_[1];
$$ LANGUAGE plperl;
Copy the code

She tests that PL/Perl is working by calling the following function:

$ psql -c 'SELECT perl_max(1, 2); ' perl_max ---------- 2 (1 row)Copy the code

Build Postgres 12

It had been a while since Cardenia had built and installed Postgres 11.8, when Postgres 12 was released. The new version looks like it has some nice new features, but Cardenia isn’t sure if she wants to upgrade from Postgres 11 in place. So she decided to install Postgres 12 in parallel with her existing Postgres 11.8.

Postgres 11.8 is installed in ~/. Local/Postgres. So where do you install the new version? Cardenia decided to include version information at the installation location to make it easier to distinguish between the two Postgres versions.

$ pg_ctl -D ~/test -l logfile stop $ mv ~/.local/postgres ~/.local/postgres-11 $ export PATH="${HOME}/.local/postgres-11/bin:${PATH}" $ pg_ctl -D ~/test -l logfile start $ psql -c 'SELECT version(); ' psql: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directoryCopy the code

Cardenia knows that the application needs LD-Linux to help binaries find dependent shared libraries at runtime, and LDD helps Cardenia print the shared libraries that binaries depend on.

$ ldd ~/.local/postgres-11/bin/psql
    linux-vdso.so.1 =>  (0x00007ffe5b9d2000)
    libpq.so.5 => not found
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6981cae000)
    libreadline.so.6 => /lib64/libreadline.so.6 (0x00007f6981a68000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f6981860000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f698155e000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f6981190000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6981eca000)
    libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f6980f66000)
Copy the code

Note: Executing AN LDD on an untrusted binary is dangerous because it executes the program being probed — see LDD Arbitrary Code Execution (catonmat.net/ldd-arbitra…

The output of the LDD shows that the dynamic linker could not find libpq.so.5. But she found out libpq.so.5 did exist by checking:

$ ls -l ~/.local/postgres-11/lib/libpq.so* lrwxrwxrwx 1 cardenia cardenia 13 Jul 23 21:14 / home/cardenia/local/postgres - 11 / lib/libpq. So - > libpq. So. 5.11 LRWXRWXRWX 1 cardenia cardenia 13 Jul 23 they / home/cardenia/local/postgres - 11 / lib/libpq. So. 5 - > libpq. So. 5.11 - RWXR - xr - x 1 cardenia cardenia 298384 Jul 23 they / home/cardenia/local/postgres - 11 / lib/libpq. So. 5.11Copy the code

Since libpq.so.5 exists, why can’t LDD find the PSQL dependency library libpq.so. Carenia decided to read the manual on dynamic linker (man 8 lD-linux.so). She learned that the linker would look for shared libraries in “/etc/ld.so.cache” and that the cache was managed by LDConfig. Unfortunately, LDConfig requires root access, so Carenia was unable to add “~cardenia/.local/postgres-11/lib” as a place to search for shared libraries. Even if she could, this configuration would apply to the entire operating system, not just her users. The alternative is to use LD_LIBRARY_PATH to search for shared libraries, but this setting applies to all programs she runs in the shell. As mentioned in the dynamic linker manual, the linker loads the shared library using the directory specified by the DR_RPATH and DT_RUNPATH attributes of the dynamic section of the binary file. Reading this, Cardenia guessed that the two attributes of PSQL were pointing to the wrong path and could not find the shared library libpq.so.5, so she used readelf to check whether PSQL had set the value of RUNPATH.

$ readelf --dynamic ~/.local/postgres-11/bin/psql
...
 0x000000000000001d (RUNPATH)            Library runpath: [/home/cardenia/.local/postgres/lib]
...
​
$ ls -l /home/cardenia/.local/postgres/lib
ls: cannot access /home/cardenia/.local/postgres/lib: No such file or directory
Copy the code

The reason was found, After the mobile installation directory (/ home/cardenia/local/postgres – > / home/cardenia/local/postgres – 11) PSQL DT_RUNPATH attribute value is still moving in front of the installation directory path (/ home/cardenia/local/postgres/lib). This causes the linker to not find libpq.so.5. This problem can be solved by rebuilding Postgres 11. But is there a way to do it without rebuilding it? The dynamic linker manual mentions that $ORIGIN allows you to use relative paths for RUNPATH, try it?

Create a relocatable Postgres- first attempt

“I’m probably not the first person to want to build a relocatable Postgres,” Cardenina says. Searching the Internet, she found PostgreSQL that matched exactly how she wanted it to be built. Because she wants to make sure that relocatable Postgres works, she sets –prefix to an unexpected final location.

CD ~ / local/SRC/postgresql 11.8 / make clean. / configure \ -- prefix = / home/cardenia/local/postgres \ - with - perl \ --disable-rpath LD_RUN_PATH='$ORIGIN/.. /lib' make -j make installCopy the code

Now that the build is installed successfully, Cardenia has moved the installation directory:

mv ~/.local/postgres ~/.local/postgres-11
Copy the code

Now try the linker to see if it can find the PSQL dependency shared library:

$ ldd ~/.local/postgres-11/bin/psql linux-vdso.so.1 => (0x00007ffcad968000) libpq.so.5 => /home/cardenia/.local/postgres-11/bin/.. /lib/libpq.so.5 (0x00007f0cc305a000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0cc2e3e000) libreadline.so.6 => /lib64/libreadline.so.6 (0x00007f0cc2bf8000) librt.so.1 => /lib64/librt.so.1 (0x00007f0cc29f0000) libm.so.6 => /lib64/libm.so.6 (0x00007f0cc26ee000) libc.so.6 => /lib64/libc.so.6 (0x00007f0cc2320000) /lib64/ld-linux-x86-64.so.2 (0x00007f0cc329d000) libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f0cc20f6000)Copy the code

Now PSQL can find the related dependency library, so now try to run PSQL statement:

​ ​

$ export PATH="${HOME}/.local/postgres-11/bin:${PATH}" $ pg_ctl -D ~/test -l logfile start waiting for server to shut down.... done server stopped $ psql -c 'SELECT version(); 'version -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- PostgreSQL 11.8 on x86_64 - PC - Linux - gnu, Compiled by GCC (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit (1 row) PSQL -c 'SELECT perl_max(1, 2); ' ERROR: could not load library "/home/cardenia/.local/postgres-11/lib/plperl.so": libperl.so: cannot open shared object file: No such file or directoryCopy the code

Failed? ! ? Cardenia decided to use LDD to see what was going on.

$ ldd ~/.local/postgres-11/lib/plperl.so linux-vdso.so.1 => (0x00007fffd4593000) libperl.so => not found libpthread.so.0  => /lib64/libpthread.so.0 (0x00007fe5c4e92000) libc.so.6 => /lib64/libc.so.6 (0x00007fe5c4ac4000) /lib64/ld-linux-x86-64.so.2 (0x00007fe5c52c4000)Copy the code

By default, libperl.so is not in the default search path of ld-linux.so:

$ find /usr -name 'libperl.so'
/usr/lib64/perl5/CORE/libperl.so
Copy the code

She cannot add the path to “/etc/ld.so.cache” without root permission, and using a single value for RUNPATH will not work. Therefore, Cardenia needs to set RUNPATH values for Postgres’s different binaries and dynamic link libraries.

Understand Postgres’s build system

Let’s look at how RUNPATH is handled in Postgres’ build system and see if we can come up with a solution for Cardenia.

Configure

Configure –enable-rpath from configure.in# l177-l182 (github.com/postgres/po…

#
# '-rpath'-like feature can be disabled
#
PGAC_ARG_BOOL(enable, rpath, yes,
              [do not embed shared library search path in executables])
AC_SUBST(enable_rpath)
Copy the code

  • PGAC_ARG_BOOL is defined in general. M4 # l77-l102 and defines two acceptable options yes and no

    – This macro is used for both –enable- and –with-

  • AC_SUBST uses the shell variable ($enable_rPATH) to create an exported variable (@enable_rpath@) in the file

  • In this case, the value of the exported variable is yes or no

Makefile

$(enable_rpath)

The only file that references @enable_rpath@ is SRC/makefile.global.in #L199

enable_rpath    = @enable_rpath@
Copy the code

Here we set a Makefile variable equal to the configure script’s shell variable enable_rPATH as follows:

$ ./configure --enable-rpath
...
$ grep '^enable_rpath' src/Makefile.global
enable_rpath    = yes
​
$ ./configure --disable-rpath
...
$ grep '^enable_rpath' src/Makefile.global
enable_rpath    = no
Copy the code

The Makefile variable $(enable_rPATH) has two references in place.

  1. src/Makefile.global.in#L526-L532

  2. src/Makefile.shlib#L196-L227

The reference in “SRC/makefile.shlib” is only used on HP-UX systems and is ignored because of our focus on Linux systems. So we focus on the reference in” src.makefile.global.in “: when $(enable_rPATH) is yes, add the link option.

# Set up rpath if enabled.  By default it will point to our libdir,
# but individual Makefiles can force other rpath paths if needed.
rpathdir = $(libdir)
​
ifeq ($(enable_rpath), yes)
LDFLAGS += $(rpath)
endif
Copy the code

$(rpath)

In the previous section, we saw that when (enable_rPATH) is set to yes,(enable\_rpath) is set to yes, and (enable_rPATH) is set to yes, the value of (rpath) is appended to the LDFLAGS variable. View all the places where rPATH is set:

$ grep --recursive '^rpath =' src/backend/utils/mb/conversion_procs/proc.mk:rpath = src/backend/snowball/Makefile:rpath = src/pl/plpgsql/src/Makefile:rpath = src/makefiles/Makefile.freebsd:rpath = -Wl,-R'$(rpathdir)' src/makefiles/Makefile.openbsd:rpath = -Wl,-R'$(rpathdir)' src/makefiles/Makefile.netbsd:rpath = -Wl,-R'$(rpathdir)' src/makefiles/Makefile.netbsd:rpath = -Wl,-R'$(rpathdir)' src/makefiles/Makefile.solaris:rpath = -Wl,-rpath,'$(rpathdir)' src/makefiles/Makefile.solaris:rpath = -Wl,-R'$(rpathdir)' src/makefiles/Makefile.linux:rpath =  -Wl,-rpath,'$(rpathdir)',--enable-new-dtagsCopy the code

You can see

  • In some places, the value of $(rpath) is set to null

  • Default values are set by platform-specific makefiles

  • The actual RPATH/RUNPATH value is set to $(rpathdir)

$(rpathdir)

As you saw in the previous section, the actual RPATH/RUNPATH value is determined by $(rpathdir), check where it is assigned:

$ grep --recursive '^rpathdir =' contrib/jsonb_plperl/Makefile:rpathdir = $(perl_archlibexp)/CORE contrib/hstore_plpython/Makefile:rpathdir = $(python_libdir) contrib/jsonb_plpython/Makefile:rpathdir = $(python_libdir)  contrib/ltree_plpython/Makefile:rpathdir = $(python_libdir) contrib/hstore_plperl/Makefile:rpathdir = $(perl_archlibexp)/CORE src/Makefile.global.in:rpathdir = $(libdir) src/pl/plperl/GNUmakefile:rpathdir = $(perl_archlibexp)/CORE src/pl/plpython/Makefile:rpathdir = $(python_libdir)Copy the code

You can see:

  • The default (rpathdir) value is (rpathdir) value is (rpathdir) value is (libdir)

  • Some build systems set $(rpathdir) to a different value

Create a relocatable Postgres- second attempt

If you want to set RPATH/RUNPATH to $ORIGIN/.. /lib, which can be implemented with the following patches:

The diff - urN postgresql - 11.8. Orig/SRC/Makefile. Global. In postgresql 11.8 / SRC/Makefile. Global. - in Postgresql 11.8. Orig/SRC/Makefile. Global. In the 2020-05-11 21:10:48. 000000000 + 0000 + + + Postgresql 11.8 / SRC/Makefile. Global. In the 2020-07-23 22:18:56. @ @ 589659501 + 0000 + 525-525, 7, 7 @ @ # Set up rpath the if enabled. By default it will point to our libdir, # but individual Makefiles can force other rpath paths if needed. -rpathdir = $(libdir) +rpathdir = $$ORIGIN/.. /lib ifeq ($(enable_rpath), yes) LDFLAGS += $(rpath)Copy the code

This patch assumes that (bindir) and (bindir) are in the same parent directory as (bindir) and (libdir).

Cardenia rebuilt Postgres 11 based on the above patch.

CD ~ / local/SRC/postgresql - 11.8. / configure \ -- prefix = / home/cardenia/local/postgres \ - with perl \ - enable -- rpath make -j make installCopy the code

Move the Postgres installation path

mv ~/.local/postgres ~/.local/postgres-11
Copy the code

This method does not override all assignments to RPATH/RUNPATH. Therefore, RUNPATH in PL/Perl in Makefile is still obtained by Perl Runtime:

rpathdir = $(perl_archlibexp)/CORE
Copy the code

Cardenia validates PSQL and plper.so in the following ways

$ readelf -d ~/.local/postgres-11/bin/psql ... 0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/../lib] ... $ ldd ~/.local/postgres-11/bin/psql linux-vdso.so.1 => (0x00007ffed39a2000) libpq.so.5 => /home/cardenia/.local/postgres-11/bin/.. /lib/libpq.so.5 (0x00007ffba7078000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffba6e5c000) libreadline.so.6 => /lib64/libreadline.so.6 (0x00007ffba6c16000) librt.so.1 => /lib64/librt.so.1 (0x00007ffba6a0e000) libm.so.6 => /lib64/libm.so.6 (0x00007ffba670c000) libc.so.6 => /lib64/libc.so.6 (0x00007ffba633e000) /lib64/ld-linux-x86-64.so.2 (0x00007ffba72bb000) libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007ffba6114000) $ readelf -d ~/.local/postgres-11/lib/plperl.so ... 0x000000000000001d (RUNPATH) Library runpath: [/usr/lib64/perl5/CORE] ... $ ldd ~/.local/postgres-11/bin/psql linux-vdso.so.1 => (0x00007ffed39a2000) libpq.so.5 => /home/cardenia/.local/postgres-11/bin/.. /lib/libpq.so.5 (0x00007ffba7078000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffba6e5c000) libreadline.so.6 => /lib64/libreadline.so.6 (0x00007ffba6c16000) librt.so.1 => /lib64/librt.so.1 (0x00007ffba6a0e000 libm.so.6 => /lib64/libm.so.6 (0x00007ffba670c000) libc.so.6 => /lib64/libc.so.6 (0x00007ffba633e000) /lib64/ld-linux-x86-64.so.2 (0x00007ffba72bb000) libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007ffba6114000)Copy the code

At this point, Cardenia finally has a relocatable Postgres 11, and PL/Perl is working.

$ export PATH="${HOME}/.local/postgres-11/bin:${PATH}" $ pg_ctl -D ~/test -l logfile start waiting for server to shut down.... done server stopped $ psql -c 'SELECT version(); ' version --------------------------------------------------------------------------------------------------------- GCC (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), compiled by GCC (GCC) 64-bit (1 row) $ psql -c 'SELECT perl_max(1, 2); ' perl_max ---------- 2 (1 row)Copy the code

By understanding executables, shared libraries, and Postgres’s build system, Cardenia can now build and install Postgres 11 and Postgres 12 in parallel. When Posting Postgres 13, she can do the same and install Postgres 13!

conclusion

Building relocatable Postgres should seem like an obvious feature. If we can do this with Patch, why doesn’t the existing Postgres build system support this feature?

Here, our patch can do this, depending on the following assumptions:

  1. We ignored non-Linux platforms such as BSD and HP-UX

  2. We assume (bindir)and(bindir) and(bindir) and(libdir) have common parent directories. But the configure script supports customization (bindir)and(bindir) and(bindir) and(libdir) for any path

  3. Our patch couldn’t successfully run tests (e.g. Installcheck and InstallCheck-world)

In fact, the Postgres build system has long proposed relocatable Postgres.

Postgres developer mailing list

In an email from PGSQL-Hackers, it was submitted to create a patch relative to RPATH/RUNPATH. The patch has several problems:

  • Check-world will fail because no executable file is installed for testing

  • Check -world successfully executed, but installCheck -world failed

  • Readelf – d. / SRC/test/isolation/isolationtester output shows that isolationtester dependent libpq. So. 5

  • Running the binary executable from the Source tree (instead of the installation directory) will fail

  • The patch submitted to the developer mailing list has the same assumption as our patch that binary and library files exist in the same parent directory (e.g. (bindir) and (bindir) and (bindir) and (libdir)).

  • For libraries in (libdir) subdirectories, libraries in (libdir) subdirectories, libraries in (libdir) subdirectories, ORIGIN/.. /lib could not find the correct dependencies for these libraries

  • It does not take effect on PGXS modules