btree_gist: Fix memory allocation formula

btree_gist: Fix memory allocation formula

This change has been suggested by the two authors listed in this commit,
both of them providing an incomplete solution (David's formula relied on
a "bytea *", while Bertrand's did not use palloc_array()).  The solution
provided in this commit uses GBT_VARKEY instead of the inconsistent
bytea for the allocation size, with a palloc_array().

The change related to Vsrt is one I am flipping to a more consistent
style, in passing.

Author: David Geier <[email protected]>
Author: Bertrand Drouvot <[email protected]>
Discussion: https://postgr.es/m/ad0748d4-3080-436e-b0bc-ac8f86a3466a@gmail.com
Discussion: https://postgr.es/m/[email protected]