ORA-12838: cannot read/modify an object after modifying it in parallel on PaaS ATP #1134

satyendra33 · 2021-04-28T12:57:18Z

I have installed utPLSQL on a PaaS ATP database which is 19c Enterprise Edition Version 19.5.0.0.0. I tried to validate installation by creating package test_betwnstr and calling exec ut.run but here I get this exception.

I added a commit at line#91 of the package UT_ANNOTATION_CACHE_MANAGER, which resolved the error, but the run is taking forever.FYI- My installation is without DDL trigger and I do not have many custom objects in schema.

ORA-06512: at "UT3.UT_ANNOTATION_CACHE_MANAGER", line 91
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 201
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 311
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 431
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 505
ORA-06512: at "UT3.UT_RUNNER", line 150

ORA-12838: cannot read/modify an object after modifying it in parallel
ORA-06512: at "UT3.UT_ANNOTATION_CACHE_MANAGER", line 91
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 201
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 311
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 431
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 505

Error starting at line : 8 in command -
begin ut.run(); end;
Error report -
ORA-12838: cannot read/modify an object after modifying it in parallel
ORA-06512: at "UT3.UT_RUNNER", line 180
ORA-06512: at "UT3.UT_ANNOTATION_CACHE_MANAGER", line 91
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 201
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 311
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 431
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 505
ORA-06512: at "UT3.UT_RUNNER", line 150
ORA-06512: at "UT3.UT", line 134
ORA-06512: at "UT3.UT", line 488
ORA-06512: at "UT3.UT", line 558
ORA-06512: at line 1
12838. 00000 - "cannot read/modify an object after modifying it in parallel"
*Cause: Within the same transaction, an attempt was made to add read or
modification statements on a table after it had been modified in parallel
or with direct load. This is not permitted.
*Action: Rewrite the transaction, or break it up into two transactions
one containing the initial modification and the second containing the
parallel modification operation.

satyendra33 · 2021-04-28T12:59:42Z

The error is gone if I add below before executing ut.run:

ALTER SESSION DISABLE PARALLEL DML;

begin ut.run(); end;

Session altered.

Between string function

Finished in .005676 seconds
0 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

jgebal · 2021-04-28T22:58:17Z

Hi @satyendra33
What version of utPLSQL are you using? Is it 3.1.10 or an older version?

jgebal · 2021-04-28T23:08:19Z

Also, is the your database using automatic degree of parallelism as per this document?
https://docs.oracle.com/en/database/oracle/oracle-database/19/vldbg/degree-parallel.html#GUID-70CAEDE1-A8D2-464A-9A0B-FDEAFBCA85C3
I'm trying to figure out how to reproduce this locally and then how can I we prevent this issue from happening in the code (exclude the framework statements from parallel execution while allowing the tested code to still use the normal DB behavior and features)

satyendra33 · 2021-04-29T05:08:21Z

This issue was produced on utPLSQL v3.1.10 . Also because it's an ATP database we did not set any parameters explicitly, all are controlled and defaulted by Oracle. I do see that parallel degree policy is set to AUTO.
Please let me know if I could be of any help.

SHOW PARAMETER PARALLEL;
NAME TYPE VALUE

awr_pdb_max_parallel_slaves integer 10
containers_parallel_degree integer 65535
fast_start_parallel_rollback string HIGH
max_datapump_parallel_per_job string 50
optimizer_ignore_parallel_hints boolean FALSE
parallel_adaptive_multi_user boolean FALSE
parallel_degree_limit string CPU
parallel_degree_policy string AUTO
parallel_execution_message_size integer 32768
parallel_force_local boolean FALSE
parallel_instance_group string
parallel_max_servers integer 24
parallel_min_degree string CPU
parallel_min_percent integer 0
parallel_min_servers integer 704
parallel_min_time_threshold string AUTO
parallel_servers_target integer 24
parallel_threads_per_cpu integer 1
recovery_parallelism integer 0

PhilippSalvisberg · 2021-04-29T05:28:16Z

@jgebal you can reproduce the issue in the current develop version when connecting as ut3_tester_helper and running the following:

alter session enable parallel dml;

begin
   ut.run('UT3$USER#:utplsql.test_user.expectations.binary.test_equal.success_on_equal_data');
end;
/

The error stack is:

ORA-12838: cannot read/modify an object after modifying it in parallel
ORA-06512: at "UT3.UT_RUNNER", line 152
ORA-06512: at "UT3.UT_ANNOTATION_CACHE_MANAGER", line 56
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 153
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 203
ORA-06512: at "UT3.UT_ANNOTATION_MANAGER", line 314
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 432
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 455
ORA-06512: at "UT3.UT_SUITE_MANAGER", line 552
ORA-06512: at "UT3.UT_RUNNER", line 139
ORA-06512: at "UT3.UT", line 134
ORA-06512: at "UT3.UT", line 488
ORA-06512: at "UT3.UT", line 623
ORA-06512: at line 2
12838. 00000 -  "cannot read/modify an object after modifying it in parallel"
*Cause:    Within the same transaction, an attempt was made to add read or
           modification statements on a table after it had been modified in parallel
           or with direct load. This is not permitted.
*Action:   Rewrite the transaction, or break it up into two transactions
           one containing the initial modification and the second containing the
           parallel modification operation.

Here's a simple variant to reproduce the error in any session:

alter session enable parallel dml;

create table t (c1 integer);

insert into t values(1);
insert into t values(2);
commit;

-- uses configured DOP
update t set c1 = c1+1;

-- throws ORA-1238
select * from t;

The last select statement throws the ORA-12838: cannot read/modify an object after modifying it in parallel.

You can solve the issue by modifying the update as follows:

update /*+noparallel */ t set c1 = c1+1;

So the fixed script looks as follows:

alter session enable parallel dml;

create table t (c1 integer);

insert into t values(1);
insert into t values(2);
commit;

-- override default DOP to ensure the changed table can be read without committing the transaction
update /*+noparallel */ t set c1 = c1+1;

-- does not throw a ORA-12838
select * from t;

I see basically two options to solve the issue in the utPLSQL framework.

a) execute alter session disable parallel dml as @satyendra33 used as workaround
b) add a noparallel hint to all DML statements in the utPLSQL framework for tables that are read after being modified.

I'd go with option b) since this tests the code in the configured way. Especially if you want to test that "your" code does not throw a ORA-12838.

jgebal · 2021-04-29T19:26:25Z

I'm having trouble reproducing this on my local
Seems like I cannot force parallel DML on my Oracle 19c slim DB docker image.
Here are my parameters:

SQL> SHOW PARAMETER PARALLEL
awr_pdb_max_parallel_slaves          integer     10
containers_parallel_degree           integer     65535
fast_start_parallel_rollback         string      HIGH
max_datapump_parallel_per_job        string      50
optimizer_ignore_parallel_hints      boolean     FALSE
parallel_adaptive_multi_user         boolean     FALSE
parallel_degree_limit                string      CPU
parallel_degree_policy               string      AUTO
parallel_execution_message_size      integer     16384
parallel_force_local                 boolean     FALSE
parallel_instance_group              string
parallel_max_servers                 integer     24
parallel_min_degree                  string      CPU
parallel_min_percent                 integer     0
parallel_min_servers                 integer     24
parallel_min_time_threshold          string      AUTO
parallel_servers_target              integer     24
parallel_threads_per_cpu             integer     8
recovery_parallelism                 integer     0
SQL>

SQL> select name,value from V$PARAMETER where name like '%cpu%';
cpu_count          
8
cpu_min_count  
8
resource_manager_cpu_allocation
8
parallel_threads_per_cpu
8
SQL>

SQL> select name,value from V$PARAMETER where name like '%parallel%';
max_datapump_parallel_per_job
50
recovery_parallelism
0
fast_start_parallel_rollback
HIGH
parallel_min_percent
0
parallel_min_servers
24
parallel_max_servers
24
parallel_instance_group

parallel_execution_message_size
16384
parallel_degree_policy
AUTO
parallel_adaptive_multi_user
FALSE
parallel_threads_per_cpu
8
parallel_min_time_threshold
1
parallel_degree_limit
CPU
parallel_force_local
FALSE
parallel_servers_target
24
optimizer_ignore_parallel_hints
FALSE
parallel_min_degree
CPU
awr_pdb_max_parallel_slaves
10
containers_parallel_degree
65535

19 rows selected.

SQL>

Here is what I got when tried reproducing with above settings on my DB.

SQL> set linesize 1000
SQL> set pagesize 0
SQL> alter session force parallel dml;

Session altered.

SQL> alter session set statistics_level=all;

Session altered.

SQL>
SQL> create table t (c1 integer);

Table created.

SQL>
SQL> insert into t values(1);

1 row created.

SQL> insert into t values(2);

1 row created.

SQL> commit;

Commit complete.

SQL>
SQL> -- uses configured DOP
SQL> update /*+ jge */ t set c1 = c1+1;

2 rows updated.

SQL>
SQL> -- throws ORA-1238
SQL> select * from t;
         2
         3

SQL>
SQL> select xplan.*
  2    from v$sql s
  3  cross apply table(dbms_xplan.display_cursor(s.sql_id, s.child_number, format => 'ALLSTATS LAST')) xplan
  4    where sql_text='update /*+ jge */ t set c1 = c1+1';
SQL_ID  a54d8d7wyhzcv, child number 0
-------------------------------------
update /*+ jge */ t set c1 = c1+1

Plan hash value: 931696821

-------------------------------------------------------------------------------------
| Id  | Operation          | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
-------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT   |      |      1 |        |      0 |00:00:00.01 |      11 |
|   1 |  UPDATE            | T    |      1 |        |      0 |00:00:00.01 |      11 |
|   2 |   TABLE ACCESS FULL| T    |      1 |      2 |      2 |00:00:00.01 |       7 |
-------------------------------------------------------------------------------------

Note
-----
   - dynamic statistics used: dynamic sampling (level=2)

SQL_ID  a54d8d7wyhzcv, child number 1
-------------------------------------
update /*+ jge */ t set c1 = c1+1

Plan hash value: 931696821

-------------------------------------------------------------------------------------
| Id  | Operation          | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
-------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT   |      |      1 |        |      0 |00:00:00.01 |      11 |
|   1 |  UPDATE            | T    |      1 |        |      0 |00:00:00.01 |      11 |
|   2 |   TABLE ACCESS FULL| T    |      1 |      2 |      2 |00:00:00.01 |       7 |
-------------------------------------------------------------------------------------

Note
-----
   - dynamic statistics used: dynamic sampling (level=2)


36 rows selected.

SQL>
SQL>
SQL> drop table t;

Table dropped.

SQL>

jgebal · 2021-04-29T19:27:34Z

I would really like to be able to reproduce this issue so that I can add a unit test case into framework test suite and therefore avoid regression in future.

PhilippSalvisberg · 2021-04-29T22:34:41Z

You should get a plan a parallel plan. Otherwise you cannot reproduce it. Something like

---------------------------------------------------
| Id  | Operation	      | Name	 | E-Rows |
---------------------------------------------------
|   0 | UPDATE STATEMENT      | 	 |	  |
|   1 |  PX COORDINATOR       | 	 |	  |
|   2 |   PX SEND QC (RANDOM) | :TQ10000 |	2 |
|   3 |    UPDATE	      | T	 |	  |
|   4 |     PX BLOCK ITERATOR | 	 |	2 |
|*  5 |      TABLE ACCESS FULL| T	 |	2 |
---------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   5 - access(:Z>=:Z AND :Z<=:Z)

Note
-----
   - dynamic statistics used: dynamic sampling (level=2)
   - automatic DOP: Computed Degree of Parallelism is 2
   - Warning: basic plan statistics not available. These are only collected when:
       * hint 'gather_plan_statistics' is used for the statement or
       * parameter 'statistics_level' is set to 'ALL', at session or system level

Can you try to create the test table with a parallel degree? Something like

create table t (c1 integer) parallel 2;

satyendra33 · 2021-04-30T07:21:38Z

@jgebal - Even I couldn't produce on my db running in VM using given method but the one below works:

insert /*+ APPEND_VALUES */
into t ( c1 ) values (1);

1 row inserted.

select * from t;

Error starting at line : 16 in command -
select * from t
Error report -
ORA-12838: cannot read/modify an object after modifying it in parallel

PhilippSalvisberg · 2021-04-30T08:12:03Z

@jgebal based on https://github.com/utPLSQL/docker-scripts/blob/main/19.3/Dockerfile#L63 the image is based on a Standard Edition 2.

Parallel DML requires Enterprise Edition. See https://docs.oracle.com/en/database/oracle/oracle-database/19/dblic/Licensing-Information.html#GUID-0F9EB85D-4610-4EDF-89C2-4916A0E7AC87

jgebal · 2021-05-21T00:00:54Z

I've build a EE docker image locally just to test it out

jgebal · 2021-05-21T00:12:09Z

I've managed to reproduce the issue and will now work on a fix.
We should probably add an EE database to our testing DB pool of we would like to be able to cover this use-case to avoid regression.

satyendra33 · 2021-05-21T03:12:56Z

@jgebal I suppose you can produce the issue without EE using below commands:

insert /*+ APPEND_VALUES */
into t ( c1 ) values (1);

1 row inserted.

select * from t;

Error starting at line : 16 in command -
select * from t
Error report -
ORA-12838: cannot read/modify an object after modifying it in parallel

jgebal mentioned this issue Nov 7, 2021

Feature/fix cache when parallel enabled #1160

Merged

jgebal closed this as completed in #1160 Nov 11, 2021

jgebal added this to the 3.1.11 milestone Nov 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ORA-12838: cannot read/modify an object after modifying it in parallel on PaaS ATP #1134

ORA-12838: cannot read/modify an object after modifying it in parallel on PaaS ATP #1134

satyendra33 commented Apr 28, 2021 •

edited

Loading

satyendra33 commented Apr 28, 2021

jgebal commented Apr 28, 2021

jgebal commented Apr 28, 2021

satyendra33 commented Apr 29, 2021

PhilippSalvisberg commented Apr 29, 2021 •

edited

Loading

jgebal commented Apr 29, 2021

jgebal commented Apr 29, 2021

PhilippSalvisberg commented Apr 29, 2021

satyendra33 commented Apr 30, 2021

PhilippSalvisberg commented Apr 30, 2021

jgebal commented May 21, 2021

jgebal commented May 21, 2021

satyendra33 commented May 21, 2021

ORA-12838: cannot read/modify an object after modifying it in parallel on PaaS ATP #1134

ORA-12838: cannot read/modify an object after modifying it in parallel on PaaS ATP #1134

Comments

satyendra33 commented Apr 28, 2021 • edited Loading

satyendra33 commented Apr 28, 2021

jgebal commented Apr 28, 2021

jgebal commented Apr 28, 2021

satyendra33 commented Apr 29, 2021

PhilippSalvisberg commented Apr 29, 2021 • edited Loading

jgebal commented Apr 29, 2021

jgebal commented Apr 29, 2021

PhilippSalvisberg commented Apr 29, 2021

satyendra33 commented Apr 30, 2021

PhilippSalvisberg commented Apr 30, 2021

jgebal commented May 21, 2021

jgebal commented May 21, 2021

satyendra33 commented May 21, 2021

satyendra33 commented Apr 28, 2021 •

edited

Loading

PhilippSalvisberg commented Apr 29, 2021 •

edited

Loading