Tomas Vondra [Fri, 21 Apr 2017 21:28:02 +0000 (23:28 +0200)]
Add the missing REFRESH MATERIALIZED VIEW to matview test suite
This adds back the missing refreshes, which were removed due to
the lack of support for a CONCURRENTLY mode. This adds back plain
refreshes (without the CONCURRENTLY keyword), which at least makes
the results the same as in upstream.
Tomas Vondra [Fri, 21 Apr 2017 20:56:14 +0000 (22:56 +0200)]
Resolve regression failures in matview test suite
All the failures are changes in query plans, switching from 2-phase
Hash Aggregate to 1-phase Group Aggregate. This seems to be happening
for two main reasons:
(1) Due to the upper planner pathification we now build additional
paths and cost them correctly. I've reviewed the plans and
the Group Aggregate paths seem cheaper than Hash Aggregate.
(2) The amount of data in the suite is rather small, so the 2-phase
aggregation seems not worth it.
Tomas Vondra [Thu, 20 Apr 2017 23:52:19 +0000 (01:52 +0200)]
Resolve trivial differences in privileges test suite
Most of the differences were due to ordering (sometimes in query but
not in the expected output) or spaces at the end of the line.
Tomas Vondra [Thu, 20 Apr 2017 23:33:39 +0000 (01:33 +0200)]
Resolve trivial failures in update regression suite
The failures were ordering-related, so simply fix the queries and
expected results.
Tomas Vondra [Thu, 20 Apr 2017 23:16:48 +0000 (01:16 +0200)]
Fix order of CREATE TABLE + INSERT for testh table in join suite
The original order (INSERT before CREATE TABLE) was obviously broken,
so correct it and also fix the expected query result.
Tomas Vondra [Thu, 20 Apr 2017 23:14:49 +0000 (01:14 +0200)]
Correct expected output for a query in join test suite
The expected output is copied from upstream. The current results
seem duplicated, for some reason.
Tomas Vondra [Thu, 20 Apr 2017 22:41:13 +0000 (00:41 +0200)]
Resolve most failures in join regression tests
This resolves most failures in 'join' regression suite. The expected
results were produced by running the new join.sql script on XL 9.5,
with a few tweaks of the explain plans (partial aggregates etc.)
Tomas Vondra [Thu, 20 Apr 2017 19:10:47 +0000 (21:10 +0200)]
Resolve most failures in rowsecurity test suite.
This resolves most failures in the rowsecurity test suite, caused
mostly by new tests from the upstream (thus the plans were missing
RemoteSubquery steps).
Tomas Vondra [Thu, 20 Apr 2017 19:05:25 +0000 (21:05 +0200)]
Propagate the distribution up when (root->distribution==NULL).
Without this patch the planner was constricting paths like this:
-> Material
-> SubquerySubPath
-> RemoteSubPath
-> BitmapHeapScan
which then led to failures to plan some of the UPDATE and DELETE
queries in updatable_views. This fixes it, and makes the planner
build the same paths as on XL 9.5 (without the RemoteSubPath,
discarding the distribution info).
This resolved all the 'could not plan distributed UPDATE/DELETE'
in updatable_views, but not universally. So either this still is
not entirely correct, or there's another independent bug.
Tomas Vondra [Sun, 19 Mar 2017 22:54:19 +0000 (23:54 +0100)]
Add missing block to 'portals' expected output
The test included block testing non-backwards-scan-capable plans
with scroll cursors, but the output was missing from the expected
file.
Tomas Vondra [Sun, 19 Mar 2017 22:50:58 +0000 (23:50 +0100)]
Remove duplicit output from 'join' test suite
The expected output contained one block twice by mistake, causing
failures when running the test suite.
Tomas Vondra [Sun, 19 Mar 2017 00:56:08 +0000 (01:56 +0100)]
Resolve failures in rules tests by adding T_CteScan to execRemote.c
The switch in determine_param_types() was missing T_CteScan case,
so some queries were failing with:
ERROR: unrecognized node type: 119
This fixes it, and it also fixes the expected output for a few other
queries in the same test suite.
Tomas Vondra [Sun, 19 Mar 2017 00:17:23 +0000 (01:17 +0100)]
Resolve failures in polymorphism suite by accepting NOTICEs
The query with sql_if()/bleat() calls is expected to produce NOTICE
messages, so just add them to the expected output.
Tomas Vondra [Sun, 19 Mar 2017 00:07:05 +0000 (01:07 +0100)]
Resolve failures in the xc_FQS regression suite
Most of the changes were simply due to plan changes introduced in
PostgreSQL 9.6, typically a SELECT DISTINCT switching from hash or
group Aggregate to Unique. The plans were verified by running the
same query on PostgreSQL 9.6, which should produce about the same
plan (except for the Remote Subquery nodes, of course).
Also fixes the result ordering for three queries by adding an
explicit ORDER BY clause.
Tomas Vondra [Sat, 18 Mar 2017 22:51:50 +0000 (23:51 +0100)]
Accept minor plan change in xl_join due to different costing
The cost of the RemoteSubplan node changed, most likely due to moving
the Sort pushdown into pathnode.c (instead of in createplan.c).
Tomas Vondra [Sat, 18 Mar 2017 20:25:59 +0000 (21:25 +0100)]
Remove 'unrecognized node type' errors from expected output
The expected output for two regression suites ("with" and "rules")
included error messages like this:
ERROR: unrecognized node type: %d
which is clearly a mistake, and just hides an error in the code.
Tomas Vondra [Sat, 18 Mar 2017 20:04:25 +0000 (21:04 +0100)]
Push recursive paths to remote nodes when construction the paths
Until now, the pushdown happened in make_recursive_union(), but that
apparently fails to update subroots (and possibly other fields), causing
segfaults in SS_finalize_plan().
Instead move the push-down logic to generate_recursion_path(), which
fixes the issue and is consistent with the overall pathification of
the planner.
With this change XL 9.6 now generates the same plans as XL 9.5.
There remains a lot to be desired, though. Firstly, it fails for queries
with nested recursive unions, because it produces a plan like this:
-> Remote Subquery Scan on all (dn1)
-> Recursive Union
-> Seq Scan on department
-> CTE Scan on x
CTE x
-> Remote Subquery Scan on all (dn1)
-> Recursive Union
-> Seq Scan on department
-> Append
-> WorkTable Scan on q
-> WorkTable Scan on x
This however fails with 'ERROR: unrecognized node type: 120' error,
because determine_param_types() in execRemote.c does not expect
T_WorkTableScan.
After resolving that issue, it however starts failing with a segfault, as
WorkTableScan nodes try to access the top recursive union node, which is
however executed in a different process, and possibly even on a different
data node (note the "Remote Subquery").
Those issues are however not resolved by this commit, and is left for
the future.
Tomas Vondra [Sat, 18 Mar 2017 19:47:32 +0000 (20:47 +0100)]
Iterate over the whole simple_rel_array in WITH RECURSIVE check
The loop over in subquery_planner() looked about like this:
for (i = 1; i < root->simple_rel_array_size - 1; i++)
which fails to check the last element in the array, due to a trivial
off-by-one error. Instead, the loop should be:
for (i = 1; i < root->simple_rel_array_size; i++)
Tomas Vondra [Sat, 18 Mar 2017 19:27:08 +0000 (20:27 +0100)]
produce the same plans on replicated tables as XL 9.5
Due to a small difference in adjust_path_distribution(), 9.6 updated
the root distribution (to path distribution) whenever it was NULL,
unlike XL 9.5.
Due to this, adjust_subplan_distribution() skipped the block that
restricts the execution on replicated tables to a single node
if (subd && !equal(subd, pathd))
{
if (IsLocatorReplicated(subd->distributionType) &&
bms_num_members(subd->restrictNodes) != 1)
{
...
}
}
which changed the generated plans from
QUERY PLAN
---------------------------------------------------------------------------
Remote Subquery Scan on any
-> Insert on base_tbl b
-> Result
SubPlan 1
-> Remote Subquery Scan on all
-> Index Only Scan using ref_tbl_pkey on ref_tbl r
Index Cond: (a = b.a)
(7 rows)
to
QUERY PLAN
---------------------------------------------------------------------------
Remote Subquery Scan on any
-> Insert on base_tbl b
-> Result
SubPlan 1
-> Remote Subquery Scan on any
-> Index Only Scan using ref_tbl_pkey on ref_tbl r
Index Cond: (a = b.a)
(7 rows)
While this plan change is mostly harmless (it merely postpones the node
selection to from planning to execution time), it generated a lot of
regression test failures.
Tomas Vondra [Sun, 5 Mar 2017 00:26:11 +0000 (01:26 +0100)]
fix failures in 'largeobject' tests due to changed role name
The role name changed from 'regresslo' to 'regress_lo_user' and one
of the source files was still using the old one.
Tomas Vondra [Sat, 4 Mar 2017 23:48:07 +0000 (00:48 +0100)]
fix failures in 'brin' regression tests
The failures seemed to be caused simply by different parameter
values, but it turned out the tests are actually broken and don't
really work as intended, for two reasons.
Firstly, the script does EXPLAIN on multiple queries and checks if
there's Bitmap Heap Scan etc. but it was only looking at the very
beginning of the result. But XL puts Remote Fast Query Execution
on top of the plan, so these checks were failing (but the WARNING
output was accepted, so no one knew).
Secondly, one of the commands populating the test table was using
subquery with ORDER BY, which is not supported in XL. So the query
was failing, and the results were a bit different from upstream.
So this commit fixes two things:
- Changes the pattern matching to look for the expected operation
anywhere in the plan. Considering that the queries are very simple
(single scan with a single condition), this is unlikely to trigger
false positives or negatives.
- Changes how the 'brintest' table is pupulated, replacing the
single subquery (with ORDER BY) with a few commands producing the
same data set as on upstream.
Tomas Vondra [Sat, 4 Mar 2017 21:48:06 +0000 (22:48 +0100)]
resolve failure in 'rowtypes' test by accepting distributed plan
The correct plan was apparently lost during the initial post-merge
conflict resolution, or something like that. So use the correct plan
from the 9.5 XL branch.
Tomas Vondra [Sat, 4 Mar 2017 21:42:08 +0000 (22:42 +0100)]
resolve the remaining failure in 'tsearch' regression suite
The ts_debug() call is not accessing any persistent data, particularly
not distributed tables. So the ORDER BY clause is not needed at all,
and removing it also makes the test closer to upstream.
Tomas Vondra [Sat, 4 Mar 2017 21:33:51 +0000 (22:33 +0100)]
resolve most issues in the 'tsearch' regression test suite
This commit addresses failures due to correlated updates and missing
support for triggers (used in upstream to update the tsearch column).
The table test_tsquery is now replicated, and instead of relyion on
triggers the column is populated explicitly.
Tomas Vondra [Sat, 4 Mar 2017 04:02:52 +0000 (05:02 +0100)]
resolve issues in create_index regression suite (obsolete error messages)
The expected output included error messages that were however made
unnecessary some time ago by making the table replicated.
Tomas Vondra [Sat, 4 Mar 2017 03:48:01 +0000 (04:48 +0100)]
fix regression failures in insert_conflict by replicating the table
The table 'insertconflict' has to be replicated, because the tests
need a unique index on (coalesce(a,0)). That is not possible on
distributed tables, because the index expression has to match the
distribution key and we only support columns there.
Tomas Vondra [Sat, 4 Mar 2017 01:17:42 +0000 (02:17 +0100)]
resolve failures in the 'alter_table' regression suite
The failures were due to foreign keys conflicting with distribution.
Resoved by replicating the tables used in the test suite.
Tomas Vondra [Sat, 4 Mar 2017 00:20:47 +0000 (01:20 +0100)]
accept plan changes due to Sort pushdown in xc_FQS_join
redistribute_path() makes sure the Sort is always pushed down, so
the plans generally change from this:
-> Sort
Output: tab3_rep.val, tab3_rep.val2
Sort Key: tab3_rep.val, tab3_rep.val2
-> Remote Subquery Scan on all
Output: tab3_rep.val, tab3_rep.val2
-> Seq Scan on public.tab3_rep
Output: tab3_rep.val, tab3_rep.val2
Filter: (tab3_rep.val > 2)
to this
-> Remote Subquery Scan on all
Output: tab3_rep.val, tab3_rep.val2
-> Sort
Output: tab3_rep.val, tab3_rep.val2
Sort Key: tab3_rep.val, tab3_rep.val2
-> Seq Scan on public.tab3_rep
Output: tab3_rep.val, tab3_rep.val2
Filter: (tab3_rep.val > 2)
The "Remote Subquery" node only shows "Sort Key" in verbose mode.
Tomas Vondra [Sat, 4 Mar 2017 00:10:23 +0000 (01:10 +0100)]
fix plan change in xl_join due to proper costing of the path
Due to accounting for the extra Sort node in redistribute_path(), the
costing now computes the costing now works properly. Originally the
Sort node was injected in createplan.c after the planning was done
(and so all the costing was done, and tweaking them would be poinless).
Tomas Vondra [Fri, 3 Mar 2017 23:03:01 +0000 (00:03 +0100)]
accept plan (added RemoteSubplan) in LockRows query ('select' suite)
Expected plan change for a new query added in the upstream, due to
adding RemoteSubplan at the top.
Tomas Vondra [Fri, 3 Mar 2017 16:08:43 +0000 (17:08 +0100)]
accept plan changes caused by Aggregate -> Partial/Finalize Aggregate
Most of the plan changes are obviously correct, caused by trivial
changes due to upstream now supporting partial aggregation. There are
a few larger changes (including e.g. switch from hash to sort), but
those are supported by also checking the query results.
Pavan Deolasee [Fri, 24 Feb 2017 10:05:24 +0000 (15:35 +0530)]
Look into the initPlans attached to lefttree of RemoteSubplan while deciding
whether correct variables are being referenced
This helps us to fix issue #81. It's not immediately clear if we should handle
this is more elegant manner than what we've done here. In PostgreSQL the
initplans are always attached to the top level plan, but in XL we add a
RemoteSubplan node on top of the top level plan. Unless we take into account
vars generated by the initPlans, we might incorrectly conclude that certain var
is not accessible by the subquery.
Pavan Deolasee [Thu, 9 Feb 2017 08:05:46 +0000 (13:35 +0530)]
Handle locking correctly in a global session.
Rearrange code so that we first check if the backend belongs to a global
sessions and if another member of the same session is holding a conflicting
lock. In such cases, the requesting backend does not wait and is granted the
lock immediately.
PG 9.6 changed a few things in this area to support locking for parallel
sessions. May be we can reuse that code in XL, but for now just use XL's code
for supporting distributed sessions
Tomas Vondra [Sun, 29 Jan 2017 18:15:29 +0000 (19:15 +0100)]
fix handling of inner/outer sortkeys in set_joinpath_distribution()
We must not reset inner/outer sortkeys unless the path is actually
redistributed. When a redistribution is not needed, the merge join
needs the original values. Otherwise this results in errors like
ERROR: inner pathkeys do not match mergeclauses
ERROR: outer pathkeys do not match mergeclauses
Tomas Vondra [Sun, 29 Jan 2017 17:45:55 +0000 (18:45 +0100)]
resolve most issues in the matview test suite
Those were fairly trivial issues - change of relation names to include
'mvtest_' prefix, ordering of SELECT results etc. There are a few
remaining differences in query plans - those may easily be correct,
but more detailed verification is needed.
Note: There seems to be quite a few differences when compared to
the matview.sql from commit
b5bce6c. Not sure why, but we probably
need to compare the tests tu upstream and minimize the difference.
Tomas Vondra [Sun, 29 Jan 2017 13:17:12 +0000 (14:17 +0100)]
fix ordering of results in the select_implicit regression test
A simple case of expected output not matching the ORDER BY.
Tomas Vondra [Sun, 29 Jan 2017 13:13:18 +0000 (14:13 +0100)]
remove GROUPING SETS from xl_limitations test for now
XL now handles grouping sets, although in a rather stupid way by
redistributing the data and executing them on the consumer node.
We may decide not to include that in 9.6 until we can distribute
(and possibly parallelize) the grouping sets, but for now let's
make the regression tests happy.
Tomas Vondra [Sun, 29 Jan 2017 02:14:36 +0000 (03:14 +0100)]
fix syntax error in foreign_key test suite
It's DISTRIBUTE BY HASH (x), not just DISTRIBUTE BY (x).
Tomas Vondra [Sun, 29 Jan 2017 02:06:51 +0000 (03:06 +0100)]
fix plans in tests broken by adding Sort Key to EXPLAIN(VERBOSE)
This fixes explain plans broken only because of the extra bit of
information added to EXPLAIN VERBOSE output. If the plan was broken
for some other reasons too, this not fix it.
Tomas Vondra [Sun, 29 Jan 2017 01:37:33 +0000 (02:37 +0100)]
fix failures due to distributed update/delete plans in some tests
Some tests are updating multiple columns of input tables, which is
not supported by XL. So instead of failing due to this limitation,
simply replicate the whole table to test whatever was the original
intent of the test.
All the UPDATE/DELETE cases fixed by this commit were failing in
XL 9.5, so none of this should be a regression.
Tomas Vondra [Sun, 29 Jan 2017 01:08:08 +0000 (02:08 +0100)]
fix breakage in 'rowsecurity' due to renamed roles used in tests
The tests originally used role names adam, bob, eve, etc. But those
roles got renamed to regress_rls_adam, regress_rls_bob etc. So fix
the exected output accordingly. Also fix some minor breakage due
to modified wording of error messages and/or data tweaks.
Tomas Vondra [Sat, 28 Jan 2017 22:43:01 +0000 (23:43 +0100)]
redistribute Limit nodes nested in MinMaxAggPath
Make sure min/max path are redistributed. Doing it in createplan.c
is probably only a temporary solution, though. First, it breaks
the costing model (the path cost does not include the redistribution),
and it also probably breaks how distribution is propagated up the
path tree.
Tomas Vondra [Sat, 28 Jan 2017 15:56:34 +0000 (16:56 +0100)]
add xl_bugs to expected/.gitignore and sql/.gitignore
Tomas Vondra [Sat, 28 Jan 2017 15:27:20 +0000 (16:27 +0100)]
add pgxc_clean/pgxc_ctl/pgxc_monitor binaries to .gitignore
Make the 'git status' a bit less verbose by ignoring the binaries,
and also th generated .c/.h files in pgxc_ctl.
Tomas Vondra [Sat, 28 Jan 2017 15:17:37 +0000 (16:17 +0100)]
remove unnecessary expected output for regression tests
There was quite a few unnecessary files in src/test/regress/expected,
not present at upstream, probably due to previous merges. As those
are most likely stale, let's just get rid of them.
Tomas Vondra [Sat, 28 Jan 2017 13:16:37 +0000 (14:16 +0100)]
minor fixes in 'cluster' regression suite (CREATE TABLE, extra .out files)
The 'CREATE TABLE clstr_tst' was wrong, as it needs to reference the
column and not the foreign key constraint. The table can't have a PK
on 'a' (does not include the distribution key), so remove that from
the expected list of constraints etc.
Also remove the unnecessary cluster_1.out and cluster_2.out files.
Tomas Vondra [Sat, 28 Jan 2017 12:55:11 +0000 (13:55 +0100)]
start both coordinators in pg_regress with is_main=false
The only difference seems to be that with is_main=false pg_regress
does not override list_addresses=*, which makes debugging of running
regression tests (e.g. when the tests get stuck) easier.
Tomas Vondra [Mon, 23 Jan 2017 00:54:30 +0000 (01:54 +0100)]
use log_line_prefix='%m ..' and log_error_verbosity='verbose' in pg_regress
This makes debugging of errors and crashes a tad easier. Timestamps
with milliseconds allow more precise correlation of events from
multiple log files (coordinators vs. datanodes). Verbose error log
messages include file and line number where the error was reported.
Tomas Vondra [Sun, 22 Jan 2017 23:51:29 +0000 (00:51 +0100)]
fix trivial failures in 'inherit' tests ('Sort Key' in EXPLAIN VERBOSE)
Tomas Vondra [Sun, 22 Jan 2017 22:57:31 +0000 (23:57 +0100)]
propagate distribution through UpperUniquePath and SetOpPath
Simply create a copy of the subpath's distribution, just like in most
other paths.
Tomas Vondra [Sun, 22 Jan 2017 22:54:39 +0000 (23:54 +0100)]
fix _readSetOp(), _readMergeAppend(), _readUnique() and _outSetOp()
Without this, the queries in 'union' test suite were crashing with
errors like "did not find '}' at end of input node" and so on. Fixed
mostly by reverting to XL 9.5 code.
Tomas Vondra [Sun, 22 Jan 2017 20:11:39 +0000 (21:11 +0100)]
properly redistribute WindowAgg input, fix plan change in window.out
We can't push down the whole WindowAgg (at least now for now), so
we need to redistribute the input properly. Luckily thanks to the
parallel aggregates, we get plans like this:
QUERY PLAN
-----------------------------------------------------------------
WindowAgg
-> Finalize GroupAggregate
Group Key: (tenk1.ten + tenk1.four)
-> Remote Subquery Scan on all (datanode_1,datanode_2)
-> Sort
Sort Key: ((tenk1.ten + tenk1.four))
-> Partial HashAggregate
Group Key: (tenk1.ten + tenk1.four)
-> Seq Scan on tenk1
(9 rows)
That however means mild breakage in window.out, due to the partial
aggregates etc.
Tomas Vondra [Sun, 22 Jan 2017 19:40:00 +0000 (20:40 +0100)]
make sure LockRows paths preserve distribution, update plans in xc_for_update
LockRows paths were not preserving the subpath's distribution, so fix
that. Also update minor plan difference (VERBOSE now shows sort key
for the redistribution step).
Tomas Vondra [Sun, 22 Jan 2017 19:17:03 +0000 (20:17 +0100)]
resolve failures in copy2 regression tests, get rid of copy2_1.out
A simple error in column name. The copy2_1.out was removed (or more
precisely, copy2.out was removed and copy2_1.out was renamed).
Tomas Vondra [Sun, 22 Jan 2017 19:11:37 +0000 (20:11 +0100)]
add space at the end of log_line_prefix in pg_regress
The GUC was missing a space at the end, so the log lines looke like
...,global_session=coord1_8874STATEMENT: ...
which is not particularly readable. With the fix it's
...,global_session=coord1_8874 STATEMENT: ...
Tomas Vondra [Sun, 22 Jan 2017 18:06:42 +0000 (19:06 +0100)]
remove the 'distribution' field from SubqueryScanPath
Considering there's a distribution field in every Path (which
the SubqueryScanPath includes), this seems unnecessary and quite
confusing, because sometimes we work with (Path*) directly, and
sometimes through the 'path' field. It took me quite a bit of
time to realize that when create_subqueryscan_path does
pathnode->distribution = X
it's not the same as doing
pathnode->path.distribution = X
Note: It's possible that we in fact need the separate field for
somehow transformed distribution?
Tomas Vondra [Sun, 22 Jan 2017 17:57:40 +0000 (18:57 +0100)]
add a redistribution missing in create_distinct_paths
A redistribution was missing in the loop handling paths already
sorted for the DISTINCT case. So add it there.
Tomas Vondra [Sun, 22 Jan 2017 17:52:20 +0000 (18:52 +0100)]
remove the comment about need for Sort in create_distinct_paths
The explicit Sort is not needed, because RemoteSubplan will take
care of that (using the built-in SimpleSort, if needed).
Tomas Vondra [Sun, 22 Jan 2017 17:36:24 +0000 (18:36 +0100)]
fix updatable_views.out to expect base_tbl to be replicated
Tomas Vondra [Sun, 22 Jan 2017 17:29:09 +0000 (18:29 +0100)]
fix width-related plan differences in xl_plan_pushdown tests
The plans only differ in width, and the new width values seem to
match the output. This was likely due to plan pushdown happening
in create_plan, after the planning is done.
Tomas Vondra [Sun, 22 Jan 2017 16:53:39 +0000 (17:53 +0100)]
accept valid distributed plans for LIMIT queries in regression tests
This typically means changing Limit->Scan and Limit->Sort plans to
Limit->Remote->Limit->Scan and Limit->Remote->Limit->Sort.
Tomas Vondra [Sun, 22 Jan 2017 16:52:18 +0000 (17:52 +0100)]
fix regression test breakage due to minor changes in output
Typically change in error message wording, description output etc.
Tomas Vondra [Sun, 22 Jan 2017 16:49:27 +0000 (17:49 +0100)]
fix ordering of results in the tsearch2 regression suite
Tomas Vondra [Sun, 22 Jan 2017 16:47:59 +0000 (17:47 +0100)]
accept valid plan changes to parallel queries in select_parallel
This typically means additional Remote Subquery Scan somewhere in
the plan.
Tomas Vondra [Sun, 22 Jan 2017 16:45:46 +0000 (17:45 +0100)]
fix tests broken by the unification of role names
A bunch of regression suites were broken by upstream changes to role
names used for testing.
Tomas Vondra [Sun, 22 Jan 2017 16:43:19 +0000 (17:43 +0100)]
add pgxc_prepared_xact results to prepared_xacts expected output
The SQL file was calling pgxc_prepared_xact(), but the results were
not included in the expected file.
Tomas Vondra [Sun, 22 Jan 2017 16:40:59 +0000 (17:40 +0100)]
fix failures in 'privileges' test suite, related to large objects
Postgres-XL currently does not support large objects, so accept the
'not supported' errors in the output.
Tomas Vondra [Sun, 22 Jan 2017 16:35:34 +0000 (17:35 +0100)]
resolve failures in 'gist' test suite by making them deterministic
As described in
e39c4afcfa, the results were platform dependent.
Tomas Vondra [Sun, 22 Jan 2017 16:31:06 +0000 (17:31 +0100)]
resolve failures in the 'foreign_key' test suite
The temporary table t2 needs to be distributed on (b) because of
the foreign key. Remove unnecessary foreign_key_1.out.
Tomas Vondra [Sun, 22 Jan 2017 16:26:34 +0000 (17:26 +0100)]
resolve failures in the 'rules' test suite
Accept the less fancy output in ruleutils tests, and remove the
rules_2.out expected output.
Tomas Vondra [Sun, 22 Jan 2017 16:22:43 +0000 (17:22 +0100)]
fix cluster regression test by properly distributing the clstr_tst
Without the DISTRIBUTE BY clause, the table was distributed by the
first column, resulting in a failure due to the foreign key.
Tomas Vondra [Sun, 22 Jan 2017 16:19:11 +0000 (17:19 +0100)]
restore expected output for groupingsets test suite from upstream
Since
44fd89e21 (and a few additional commits), XL should generate
correct paths for grouping sets instead of rejecting them as
unsupported feature. So restore the expected output from upstream
as XL should produce the same results.
Tomas Vondra [Sun, 22 Jan 2017 16:14:12 +0000 (17:14 +0100)]
fix foreign_data regression tests, remove foreign_data_1.out
As XL does not support FDW, the resolution was mostly about adding
the 'does not exist' errors as narrower table listings as expected.
Also remove the foreign_data_1.out, mostly for the same reasons as
arrays_1.out (unnecessary and stale).
Tomas Vondra [Sun, 22 Jan 2017 16:03:34 +0000 (17:03 +0100)]
minor fixes in 'arrays' regression tests
The ORDER BY is not needed, because arr_tbl is replicated and so the
output is as stable as on regular PostgreSQL.
Remove arrays_1.out, as it's stale and unnecessary (arrays.out is
used anyway).
Tomas Vondra [Fri, 20 Jan 2017 23:55:23 +0000 (00:55 +0100)]
fix output of json_agg(), remove unused json_agg_* prototypes etc.
json_agg_transfn() was relying on json_agg_collectfn() to generate
the initial '[' for the output, but that only worked with the old
XL-specific implementation. With the new implementation adopted from
upstream, this lead to output like this:
{"x":1},{"x":2}]
Fixed by removing the ifdef in json_agg_transfn(). While doing that,
I've noticed the json_agg_collectfn() is still in json.c, although
it's not needed anymore, and similarly for json_agg_in/out prototypes.
Tomas Vondra [Fri, 20 Jan 2017 23:34:46 +0000 (00:34 +0100)]
fix 'ERROR: unrecognized token' failures in _readWindowAgg()
The _outWindowAgg() and _readWindowAgg() were inconsistent, resulting
in failures in stringToNode(). In particular _outWindowAgg() was
producing output with tokens :partOperations and :ordOperations, while
_readWindowAgg() expected :partOperators and :ordOperators.
Perhaps more importantly, _readWindowAgg() used READ_OID_ARRAY marco
to parse :ordOperators, but the macro ignores portable_output flag.
So parsing the string failed.
Fixed mostly by reverting to the XL 9.5 code.
Tomas Vondra [Fri, 20 Jan 2017 23:18:41 +0000 (00:18 +0100)]
enforce sorting when RemoteSubplan feeds data to GroupAggregate
When the second step of distributed aggregation is executed using
GroupAggregate, the RemoteSubplan needs to produce sorted output.
We might insert Sort node above the RemoteSubplan, but we don't
do that as we wan to push as much work to the remote side.
Therefore the remote part of the plan needs to produce sorted
output, and RemoteSubplan needs to perform merge-sort. For the
code generating remote parallel paths, this was already done,
because the code injects an explicit sort on top of the Gather
when the second phase is GroupAggregate.
But for the special XL plans, using partial aggregates without
the Gather, this was not happening, so when the partial path was
executed using HashAggregate, the RemoteSubplan did not know
it needs to enforce sorted output.
Fixed by adding an explicit Sort node as the last step of the
remote path.
Tomas Vondra [Fri, 20 Jan 2017 22:46:23 +0000 (23:46 +0100)]
display sort keys for 'Remote Subquery Scan' nodes in EXPLAIN VERBOSE
This makes analysis of query plans a bit easier, as it shows which
remote plans may be sorted automatically (using merge-sort built
into the remote subplan). Only shows this for VERBOSE output though,
because in most cases it's obvious how the data is sorted.
Tomas Vondra [Fri, 20 Jan 2017 15:02:17 +0000 (16:02 +0100)]
fix the segfault crashes in
6de274a02
In short, create_remotescan_plan() was a bit confused when adding
the explicit Sort node, passing it the remote subplan node to
prepare_sort_from_pathkeys() while instead it should pass it the
subplan.
That resulted in building remote plans like this:
{REMOTESUBPLAN :startup_cost 188.23 :total_cost 243.52 ...
:targetlist ({TARGETENTRY :expr {VAR ...}}
{TARGETENTRY :expr {VAR ...}}
{TARGETENTRY :expr {OPEXPR ...}})
:lefttree {SORT :startup_cost 88.23 :total_cost 91.41 ...
:targetlist ({TARGETENTRY :expr {VAR ...}}
{TARGETENTRY :expr {VAR ...}})
:lefttree {APPEND ... }
...}
:distributionType H
:distributionKey 3
:distributionNodes (i 0 1)
:distributionRestrict (i 0 1)
:nodeList (i 0 1)
:execOnAll true
:sort {SIMPLESORT ...}
:cursor p_1_532c_2d :unique 0
}
Notice the Sort has only two targetlist entries, but distribution
key is 3. Which then triggers the segfault in PortalStart, when
accessing a non-existent target entry.
Fixed by making create_remotescan_plan sane again, and also fix
parameters passed to redistribute_path in set_joinpath_distribution,
although only in block wrapped in NOT_USED.
Also add a simple Assert() into PortalStart() to make detection of
such issues easier, instead of having to investigate segfaults.
Tomas Vondra [Fri, 20 Jan 2017 03:18:10 +0000 (04:18 +0100)]
WIP: mark Sort push-down (through RemoteSubplan) work again correctly
After moving the path redistribution and various bits to pathnode.c,
the explicit Sort nodes needed by Merge Joins with unsorted paths
were not properly pushed to the remote node (but on top of it).
Fixed by a bit of hacking set_joinpath_distribution(). Admittedly,
set_joinpath_distribution() is a mighty beast that would benefit
from a bit of refactoring.
This fixes most of the EXPLAIN regressions, although a bunch of
plans are still slightly different. But that seems to be either
because we managed to find a different (and slightly cheaper)
plan, or due to some other bug (likely in distribution handling).
FIXME This however seems to break portals/transactions regression
tests, resulting in strange crashes in PortalStart due to access
beyond the end of tupDesc->attrs:
#0 PortalStart (portal=0x124ddf0, params=0x0, eflags=0, snapshot=0x0) at pquery.c:791
791 InvalidOid :
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.22-18.fc23.x86_64
(gdb) bt
#0 PortalStart (portal=0x124ddf0, params=0x0, eflags=0, snapshot=0x0) at pquery.c:791
#1 0x00000000008720e5 in exec_bind_message (input_message=0x7ffec76f1790) at postgres.c:2217
#2 0x000000000087566f in PostgresMain (argc=1, argv=0x122f768, dbname=0x122f408 "regression", username=0x122f3e8 "user") at postgres.c:4788
#3 0x00000000007e7d63 in BackendRun (port=0x1255ae0) at postmaster.c:4496
#4 0x00000000007e74c7 in BackendStartup (port=0x1255ae0) at postmaster.c:4170
#5 0x00000000007e3887 in ServerLoop () at postmaster.c:1806
#6 0x00000000007e2ebc in PostmasterMain (argc=9, argv=0x122e390) at postmaster.c:1414
#7 0x00000000007103a0 in main (argc=9, argv=0x122e390) at main.c:228
(gdb) list
786
787 /*
788 * Set up locator if result distribution is requested
789 */
790 keytype = queryDesc->plannedstmt->distributionKey == InvalidAttrNumber ?
791 InvalidOid :
792 queryDesc->tupDesc->attrs[queryDesc->plannedstmt->distributionKey-1]->atttypid;
793 locator = createLocator(
794 queryDesc->plannedstmt->distributionType,
795 RELATION_ACCESS_INSERT,
(gdb) print queryDesc->plannedstmt->distributionKey
$1 = 3
(gdb) print queryDesc->tupDesc->natts
$2 = 2
In transaction.sql, this seems to be triggered by the 'revalidate_bug'
test case starting at like 378.
Tomas Vondra [Fri, 20 Jan 2017 03:11:06 +0000 (04:11 +0100)]
fix parsing of grpColIdx in _readGroup()
Parsing of :grpColIdx got broken during the merge conflict resolution.
Note: I don't see the bug, though. Reverting to code from XL 9.5 fixed
the issue (failing to find '}' at the end of the node) for me, but
maybe that's a fluke. So perhaps look at this again.
Tomas Vondra [Fri, 20 Jan 2017 01:34:39 +0000 (02:34 +0100)]
comment about a place adding unnecessary top-level RemoteSubpans
Tomas Vondra [Fri, 20 Jan 2017 01:15:02 +0000 (02:15 +0100)]
make sure explicit Sorts in Merge Append are pushed to remote nodes
When redistributing subpaths of Merge Append, an explicit sort may be
required. In that case we want to push it to the remote node instead
of performing it locally.
This was originally done in create_remotescan_plan(), i.e. after all
the planning was done. After the upper-planner pathification that's
not very elegant, so this patch moves this logic to pathnode.c.
The design follows what create_merge_append_path was already doing,
we only do the costing in redistribute_path() and the Sort node is
still construted within create_plan().
Tomas Vondra [Fri, 20 Jan 2017 01:00:13 +0000 (02:00 +0100)]
add _readSimpleSort() back to readfuncs.c (probably removed during merge)
Tomas Vondra [Thu, 19 Jan 2017 21:04:06 +0000 (22:04 +0100)]
make output of 'select' test suite stable by adding ORDER BY
In XL the order of rows may change depending on timing of responses
from remote nodes, unless the query has an explicit ORDER BY (on
PostgreSQL the regression tests often rely on reading the data from
disk in a single process).
Tomas Vondra [Thu, 19 Jan 2017 20:51:05 +0000 (21:51 +0100)]
remove 'missing redistribute_path' error from recurse_set_operations
I came to the conclusion that we probably don't need to redistribute
the path at this place. We're producing a generic subquery path with
particular distribution - if if a path with different distribution
is needed, it should be responsibility of the caller to do that.
The assumption is that there might be multiple possible plans, each
requiring a path with a different distribution.
Tomas Vondra [Thu, 19 Jan 2017 16:18:20 +0000 (17:18 +0100)]
obvious plan changes in inherit, aggregates, select and select_distinct
Update expected regression results in case of obvious plan changes,
as for example converting 2-phase aggregation to Final/Partial,
or wrapping the whole query into Remote Fast Query.
Tomas Vondra [Thu, 19 Jan 2017 14:47:01 +0000 (15:47 +0100)]
resolve most differences in insert_conflict regression test
The plan changes are either due to parallel query, or due to adding
a RemoteSubplan node at the top of the plan.
Tomas Vondra [Thu, 19 Jan 2017 14:30:06 +0000 (15:30 +0100)]
resolve most differences in create_index test suite
Most of the differences were trivial consequence of using partial
aggregates, instead of the XL-specific 2-phase aggregate. The
main difference is that that EXPLAIN now prefixes the Aggregate
nodes with either "Finalize" or "Partial" (while before it used
just Aggregate).
There's one remaining plan change, but that also switches from
plain Index Scan to Bitmap Index Scan. That probably deserves
a bit more attention and further investigation.
Tomas Vondra [Thu, 19 Jan 2017 13:44:02 +0000 (14:44 +0100)]
resolve failures in box regression tests (missing SP-GiST part)
The expected output was missing the new section, testing SP-GiST
on a box column. Make sure the table with test data is replicated
to get stable output.
Tomas Vondra [Thu, 19 Jan 2017 13:41:17 +0000 (14:41 +0100)]
resolve failures in timestamp/timestamptz regression tests
Update the SQL and expected output for XL, and remove unnecessary
whitespace (mostly at the end of line) to make diffing easier.
Tomas Vondra [Thu, 19 Jan 2017 13:08:37 +0000 (14:08 +0100)]
use text argument instead of cstring in pg_msgmodule_set()
opr_sanity checks that functions with cstring arguments are either
type input or conversion functions. Tweaking the function to accept
text seems like a better idea than relaxing opr_sanity tests.
Tomas Vondra [Thu, 19 Jan 2017 12:18:45 +0000 (13:18 +0100)]
mark pg_msgmodule_disable_all() as PROVOLATILE_IMMUTABLE
The pg_proc entry had provolatile=0, which is not a legal value,
so it was triggering an error in opr_sanity regression. Mark the
function as PROVOLATILE_IMMUTABLE.
Tomas Vondra [Thu, 19 Jan 2017 09:40:01 +0000 (10:40 +0100)]
skip RenameSequenceGTM() for SET SCHEMA with the same schema
Commit
bc4996e6 in upstream changed how CheckSetNamespace() handles
moving to the same namespace, e.g. when executing
ALTER SEQUENCE s1.s SET SCHEMA s1;
On PostgreSQL 9.5 this fails with an error, but since
bc4996e6 the
command is silently ignored (which was already the case for other
object types - ALTER EXTENSION etc).
XL however relied on the ERROR interrupting the program flow before
the call to RenameSequenceGTM(), so after
bc4996e6 it was executed
anyway.
GTM however does not expect unnecessary sequence renames, and treats
this as an error in seq_add_seqinfo.
Tomas Vondra [Thu, 19 Jan 2017 07:56:07 +0000 (08:56 +0100)]
add name of the originating node to application_name
Until now the application_name was set to 'pgxc' for all connections.
Set it to 'pgxc:NODE_NAME' instead, to make it easier to understand
data from pg_stat_activity.
Tomas Vondra [Wed, 18 Jan 2017 15:56:12 +0000 (16:56 +0100)]
fix reading of :mergeCollations in _readMergeJoin()
Fixes error observed in many regression tests
ERROR: did not find '}' at end of input node
Tomas Vondra [Wed, 18 Jan 2017 15:29:15 +0000 (16:29 +0100)]
properly decide when Grouping Sets do not require redistribution
When adding a GroupingSetsPath, an explicit redistribution is not
needed when either (distribution=NULL) or for replicated tables.
This resolves the crash in 'groupingsets' regression set, due to
triggering this assert in make_remotesubplan():
Assert(!equal(resultDistribution, execDistribution))
Tomas Vondra [Wed, 18 Jan 2017 14:59:29 +0000 (15:59 +0100)]
do not use Append with redistributed childrels as a partial path
Currently the subpaths are redistributed in create_append_path,
but even though that sets path.parallel_safe=false, it's too late
for set_append_rel_pathlist() to realize that.
It simply grabs the first path from childrel->partial_pathlist and
builds an AppendPath on top of that, assuming the resulting Append
is also parallel_safe.
But if the child relations had to be redistributed by adding a
RemoteSubplan on each of them, then the Append is parallel unsafe
and the attempt to add Gather on top of that fails because of
tripping on Assert(subpath->parallel_safe).
Make sure the result of create_append_path is also parallel safe
before adding it as a partial path.
I'd be surprised if there were no other places relying on the same
assumption. Ultimately, we probably want to build the redistributed
paths before calling create_append_path().
Tomas Vondra [Wed, 18 Jan 2017 11:32:42 +0000 (12:32 +0100)]
remove an extra lappend(subroots, subroot) from inheritance_planner
This resolves Assert() failure in create_modifytable_path(), causing
failures in regression tests (inherit, updatable_views, alter_table,
returning).
Tomas Vondra [Tue, 17 Jan 2017 20:06:37 +0000 (21:06 +0100)]
resolve crashes in queries involving merge-sort in a RemoteSubplan