Updated readme for installation with trigger.

Added additional tests for annotation management with / without trigger. Fixed code for situations when trigger gets added on pre-existing tests.
utPLSQL · jgebal · Jun 10, 2019 · Mar 15, 2019 · Mar 29, 2019 · Mar 29, 2019
commit 94a1c23389a955fc193c7054bc57f97799363233
diff --git a/docs/userguide/install.md b/docs/userguide/install.md
@@ -72,16 +72,31 @@ The utPLSQL may be installed on any supported version of Oracle Database [see](h
 * 12c
 * 12c R2
 * 18c
+* 19c
 
 # Headless installation
 
-To install the utPLSQL into a new database schema and grant it to public, execute the script `install_headless.sql` as SYSDBA.
+utPLSQL can be installed with DDL trigger, to enable tracking of DDL changes to your unit test packages.
+This is the recommended installation approach, when you want to compile and run unit test packages in a schema containing huge amount of database packages (for example Oracle EBS installation schema).
+The reason for having DDL trigger is to enable in-time annotation parsing for utPLSQL.
+Without DDL trigger, utPLSQL needs to investigate your schema objects last_ddl_timestamp each time tests are executed to check if any of DB packages were changed in given schema and if they need scanning for annotation changes.
+This process can be time-consuming if DB schema is large.     
 
-The script accepts three optional parameters that define:
+The headless scripts accept three optional parameters that define:
 - username to create as owner of utPLSQL (default `ut3`)  
 - password for owner of utPLSQL (default `XNtxj8eEgA6X6b6f`)
 - tablespace to use for storage of profiler data (default `users`)  
 
+The scripts need to be executed by `SYSDBA`, in order to grant access to `DBMS_LOCK` and `DBMS_CRYPTO` system packages.
+
+*Note:* Grant on `DBMS_LOCK` is required only for installation on Oracle versions below 18c. For versions 18c and above, utPLSQL uses `DBMS_SESSION.SLEEP` so access to `DBMS_LOCK` package is no longer needed. 
+
+*Note:* The user performing the installation must have the `ADMINISTER DATABASE TRIGGER` privilege. This is required for installation of trigger that is responsible for parsing annotations at at compile-time of a package.
+
+## Installation without DDL trigger
+
+To install the utPLSQL into a new database schema and grant it to public, execute the script `install_headless.sql` as SYSDBA.
+
 Example invocation of the script from command line:
 ```bash
 cd source
@@ -94,16 +109,37 @@ cd source
 sqlplus sys/sys_pass@db as sysdba @install_headless.sql utp3 my_verySecret_password utp3_tablespace   
 ```
 
-The script needs to be executed by `SYSDBA`, in order to grant access to `DBMS_LOCK` and `DBMS_CRYPTO` system packages.
+## Installation with DDL trigger
+
+To install the utPLSQL into a new database schema and grant it to public, execute the script `install_headless_with_trigger.sql` as SYSDBA.
+
+Example invocation of the script from command line:
+```bash
+cd source
+sqlplus sys/sys_pass@db as sysdba @install_headless_with_trigger.sql  
+```
+
+Invoking script with parameters:
+```bash
+cd source
+sqlplus sys/sys_pass@db as sysdba @install_headless_with_trigger.sql utp3 my_verySecret_password utp3_tablespace   
+```
+
+*Note:* 
 
-*Note:* Grant on `DBMS_LOCK` is required on Oracle versions below 18c
+When installing utPLSQL into database with existing unit test packages, utPLSQL will not be able to already-existing unit test packages.
+When utPSLQL was installed with DDL trigger, you have to do one of:
+- Recompile existing Unit Test packages to make utPLSQL aware of their existence 
+- Invoke `exec ut_runner.rebuild_annotation_cache(a_schema_name=> ... );` for every schema containing unit tests in your database
 
+Steps above are required to assure annotation cache is populated properly from existing objects.
+Rebuilding annotation cache might be faster than code recompilation.     
 
 # Recommended Schema
 It is highly recommended to install utPLSQL in it's own schema. You are free to choose any name for this schema.
 Installing uPLSQL into shared schema is really not recommended as you loose isolation of framework.
 
-If the installation and utPLSQL owner user is one and the same, the user must have the following Oracle system permissions before you can proceed with the installation.
+If the installing user and utPLSQL owner is one and the same, the user must have the following Oracle system permissions before you can proceed with the installation.
 
   - CREATE SESSION
   - CREATE PROCEDURE
@@ -113,6 +149,7 @@ If the installation and utPLSQL owner user is one and the same, the user must ha
   - CREATE VIEW
   - CREATE SYNONYM
   - ALTER SESSION
+  - CREATE TRIGGER
 
 In addition the user must be granted the execute privilege on `DBMS_LOCK` and `DBMS_CRYPTO` packages.
 
@@ -121,11 +158,9 @@ The uninstall process will **not** drop profiler tables, as they can potentially
 
 It is up to DBA to maintain the storage of the profiler tables.
 
-Additionally the user performing the installation must have a `ADMINISTER DATABASE TRIGGER` privilege. This is required for installation of trigger that is responsible for parsing annotations at at compile-time of a package. 
-
 # Manual installation procedure
 
-### Creating schema for utPLSQL
+## Creating schema for utPLSQL
 To create the utPLSQL schema and grant all the required privileges execute script `create_utplsql_owner.sql` from the `source` directory with parameters:
 
   - `user name` - the name of the user that will own of utPLSQL object
@@ -138,8 +173,8 @@ cd source
 sqlplus sys/sys_password@database as sysdba @create_utPLSQL_owner.sql ut3 ut3 users  
 ```
 
-### Installing utPLSQL
-To install the utPLSQL framework into your database run the `/source/install.sql` script and provide `schema_name` where utPLSQL is to be installed.  
+## Installing utPLSQL
+To install the utPLSQL framework into your database, go to `source` directory, run the `install.sql` providing the `schema_name` for utPLSQL as parameter.  
 Schema must be created prior to calling the `install` script.
 You may install utPLSQL from any account that has sufficient privileges to create objects in other users schema.  
 
@@ -149,7 +184,25 @@ cd source
 sqlplus admin/admins_password@database @install.sql ut3  
 ```
 
-### Allowing other users to access the utPLSQL framework
+## Installing DDL trigger
+To minimize startup time of utPLSQL framework (especially on a database with large schema) it is recommended to install utPLSQL DDL trigger to enable utPLSQL annotation to be updated at compile-time.
+
+It's recommended to install DDL trigger when connected as `SYSDBA` user. Trigger is created in utPLSQL schema.
+If using the owner schema of utPLSQL to install trigger, the owner needs to have `ADMINISTER DATABASE TRIGGER` and `CREATE TRIGGER` system privileges. 
+If using different user to install trigger, the user needs to have `ADMINISTER DATABASE TRIGGER` and `CREATE ANY TRIGGER` system privileges. 
+
+To install DDL trigger go to `source` directory, run the `install_ddl_trigger.sql` providing the `schema_name` for utPLSQL as parameter.
+
+Example invocation:
+```bash
+cd source
+sqlplus admin/admins_password@database @install_ddl_trigger.sql ut3  
+```
+
+*Note:* Trigger can be installed ant any point in time.
+
+
+## Allowing other users to access the utPLSQL framework
 In order to allow other users to access utPLSQL, synonyms must be created and privileges granted.
 You have two options:
 

diff --git a/source/check_sys_grants.sql b/source/check_sys_grants.sql
@@ -1,8 +1,6 @@
+DEFINE expected_grants = "&1"
 declare
-  c_expected_grants constant dbmsoutput_linesarray
-  := dbmsoutput_linesarray(
-      'CREATE TYPE','CREATE VIEW','CREATE SYNONYM','CREATE SEQUENCE','CREATE PROCEDURE','CREATE TABLE', 'ADMINISTER DATABASE TRIGGER'
-  );
+  c_expected_grants constant dbmsoutput_linesarray := dbmsoutput_linesarray( &expected_grants );
 
   l_expected_grants dbmsoutput_linesarray := c_expected_grants;
   l_missing_grants varchar2(4000);

diff --git a/source/core/annotations/ut_annotation_manager.pkb b/source/core/annotations/ut_annotation_manager.pkb
@@ -51,30 +51,34 @@ create or replace package body ut_annotation_manager as
     return l_result;
   end;
 
-  function get_annotation_objs_info(a_object_owner varchar2, a_object_type varchar2, a_parse_date timestamp := null) return ut_annotation_objs_cache_info is
-    l_rows         sys_refcursor;
-    l_ut_owner     varchar2(250) := ut_utils.ut_owner;
-    l_objects_view varchar2(200) := ut_metadata.get_objects_view_name();
-    l_cursor_text  varchar2(32767);
-    l_result       ut_annotation_objs_cache_info;
-    l_object_owner varchar2(250);
-    l_object_type  varchar2(250);
+  function get_annotation_objs_info(
+    a_object_owner varchar2,
+    a_object_type  varchar2,
+    a_parse_date   timestamp := null,
+    a_full_scan    boolean   := true
+  ) return ut_annotation_objs_cache_info is
+    l_rows          sys_refcursor;
+    l_ut_owner      varchar2(250) := ut_utils.ut_owner;
+    l_objects_view  varchar2(200) := ut_metadata.get_objects_view_name();
+    l_cursor_text   varchar2(32767);
+    l_result        ut_annotation_objs_cache_info;
+    l_object_owner  varchar2(250);
+    l_object_type   varchar2(250);
   begin
     ut_event_manager.trigger_event(
-      'get_annotation_objs_info - start ( ut_trigger_check.is_alive = '
-      || case when ut_trigger_check.is_alive() then 'Y' else 'N' end || ' )'
+      'get_annotation_objs_info - start ( a_full_scan = ' || ut_utils.to_string(a_full_scan) || ' )'
     );
-    if ut_trigger_check.is_alive() then
+    if not a_full_scan then
       l_cursor_text :=
         q'[select ]'||l_ut_owner||q'[.ut_annotation_obj_cache_info(
-                      object_owner => i.object_owner,
-                      object_name => i.object_name,
-                      object_type => i.object_type,
+                      object_owner  => i.object_owner,
+                      object_name   => i.object_name,
+                      object_type   => i.object_type,
                       needs_refresh => 'N'
                     )
              from ]'||l_ut_owner||q'[.ut_annotation_cache_info i
             where i.object_owner = :a_object_owner
-              and i.object_type = :a_object_type]';
+              and i.object_type  = :a_object_type]';
       open l_rows for l_cursor_text  using a_object_owner, a_object_type;
     else
       if a_object_owner is not null then
@@ -85,17 +89,17 @@ create or replace package body ut_annotation_manager as
       end if;
       l_cursor_text :=
         q'[select ]'||l_ut_owner||q'[.ut_annotation_obj_cache_info(
-                      object_owner => o.owner,
-                      object_name => o.object_name,
-                      object_type => o.object_type,
+                      object_owner  => o.owner,
+                      object_name   => o.object_name,
+                      object_type   => o.object_type,
                       needs_refresh => case when o.last_ddl_time < cast(i.parse_time as date) then 'N' else 'Y' end
                     )
              from ]'||l_objects_view||q'[ o
              left join ]'||l_ut_owner||q'[.ut_annotation_cache_info i
-               on o.owner = i.object_owner
+               on o.owner       = i.object_owner
               and o.object_name = i.object_name
               and o.object_type = i.object_type
-            where o.owner = ']'||l_object_owner||q'['
+            where o.owner       = ']'||l_object_owner||q'['
               and o.object_type = ']'||l_object_type||q'['
               and ]'
           || case
@@ -105,7 +109,7 @@ create or replace package body ut_annotation_manager as
              end;
       open l_rows for l_cursor_text  using a_parse_date;
     end if;
-    fetch l_rows bulk collect into l_result limit 1000000;
+    fetch l_rows bulk collect into l_result limit 10000000;
     close l_rows;
     ut_event_manager.trigger_event('get_annotation_objs_info - end (count='||l_result.count||')');
     return l_result;
@@ -207,14 +211,15 @@ create or replace package body ut_annotation_manager as
   end;
 
 
-  procedure rebuild_annotation_cache(
+  procedure refresh_annotation_cache(
     a_object_owner varchar2,
     a_object_type  varchar2,
     a_info_rows    ut_annotation_objs_cache_info
   ) is
     l_objects_to_parse       ut_annotation_objs_cache_info;
   begin
-    select value(x)bulk collect into l_objects_to_parse
+    select value(x)
+      bulk collect into l_objects_to_parse
       from table(a_info_rows) x where x.needs_refresh = 'Y';
 
     ut_event_manager.trigger_event('rebuild_annotation_cache - start (l_objects_to_parse.count = '||l_objects_to_parse.count||')');
@@ -243,12 +248,10 @@ create or replace package body ut_annotation_manager as
   --public definitions
   ------------------------------------------------------------
   procedure rebuild_annotation_cache(a_object_owner varchar2, a_object_type varchar2) is
+    l_annotation_objs_info ut_annotation_objs_cache_info;
   begin
-    rebuild_annotation_cache(
-      a_object_owner,
-      a_object_type,
-      get_annotation_objs_info(a_object_owner, a_object_type, null)
-    );
+    l_annotation_objs_info := get_annotation_objs_info(a_object_owner, a_object_type, null, true);
+    refresh_annotation_cache( a_object_owner, a_object_type, l_annotation_objs_info );
   end;
 
   procedure trigger_obj_annotation_rebuild is
@@ -273,9 +276,23 @@ create or replace package body ut_annotation_manager as
         select a_object_name as name, column_value||chr(10) as text from table(l_sql_lines);
       return l_result;
     end;
-  begin
-    ut_trigger_check.is_alive();
 
+    function get_source_for_object(a_object_owner varchar2, a_object_name varchar2, a_object_type varchar2) return sys_refcursor is
+      l_result       sys_refcursor;
+      l_sources_view varchar2(200) := ut_metadata.get_source_view_name();
+    begin
+      open l_result for
+        q'[select :a_object_name, s.text
+             from ]'||l_sources_view||q'[ s
+            where s.type  = :a_object_type
+              and s.owner = :a_object_owner
+              and s.name  = :a_object_name
+          order by s.line]'
+        using a_object_name, a_object_type, a_object_owner, a_object_name;
+      return l_result;
+    end;
+
+  begin
     if ora_dict_obj_type in ('PACKAGE','PROCEDURE','FUNCTION','TYPE') then
 
       l_object_to_parse := ut_annotation_obj_cache_info(ora_dict_obj_owner, ora_dict_obj_name, ora_dict_obj_type, 'Y');
@@ -287,27 +304,29 @@ create or replace package body ut_annotation_manager as
           get_source_from_sql_text(ora_dict_obj_name, l_sql_text, l_parts)
         );
       elsif ora_sysevent = 'ALTER' then
-        null;
-        --update parse_time
+        build_annot_cache_for_sources(
+          ora_dict_obj_owner, ora_dict_obj_type,
+          get_source_for_object(ora_dict_obj_owner, ora_dict_obj_name, ora_dict_obj_type)
+          );
       elsif ora_sysevent = 'DROP' then
         ut_annotation_cache_manager.remove_from_cache(ut_annotation_objs_cache_info(l_object_to_parse));
       end if;
     end if;
   end;
 
   function get_annotated_objects(a_object_owner varchar2, a_object_type varchar2, a_parse_date timestamp := null) return ut_annotated_objects pipelined is
-    l_info_rows              ut_annotation_objs_cache_info;
+    l_annotation_objs_info   ut_annotation_objs_cache_info;
     l_cursor                 sys_refcursor;
     l_results                ut_annotated_objects;
     c_object_fetch_limit     constant integer := 10;
+    l_full_scan_needed       boolean := not ut_trigger_check.is_alive();
   begin
     ut_event_manager.trigger_event('get_annotated_objects - start');
-
-    l_info_rows := get_annotation_objs_info(a_object_owner, a_object_type, a_parse_date);
-    rebuild_annotation_cache(a_object_owner, a_object_type, l_info_rows);
+    l_annotation_objs_info := get_annotation_objs_info(a_object_owner, a_object_type, a_parse_date, l_full_scan_needed);
+    refresh_annotation_cache(a_object_owner, a_object_type, l_annotation_objs_info);
 
     --pipe annotations from cache
-    l_cursor := ut_annotation_cache_manager.get_annotations_for_objects(l_info_rows, a_parse_date);
+    l_cursor := ut_annotation_cache_manager.get_annotations_for_objects(l_annotation_objs_info, a_parse_date);
     loop
       fetch l_cursor bulk collect into l_results limit c_object_fetch_limit;
       for i in 1 .. l_results.count loop
@@ -357,5 +376,6 @@ create or replace package body ut_annotation_manager as
     end if;
     return l_result;
   end;
+
 end ut_annotation_manager;
 /
diff --git a/source/core/annotations/ut_annotation_manager.pks b/source/core/annotations/ut_annotation_manager.pks
@@ -56,7 +56,10 @@ create or replace package ut_annotation_manager authid current_user as
    */
   procedure purge_cache(a_object_owner varchar2, a_object_type varchar2);
 
-
+
+  /*
+  * Returns a hash value of suitepath based on input path and random seed
+  */
   function hash_suite_path(a_path varchar2, a_random_seed positiven) return varchar2;
 
 end ut_annotation_manager;

diff --git a/source/core/annotations/ut_trigger_annotation_parsing.trg b/source/core/annotations/ut_trigger_annotation_parsing.trg
@@ -2,17 +2,19 @@ create or replace trigger ut_trigger_annotation_parsing
   after create or alter or drop
 on database
 begin
-  if   ( ora_dict_obj_type in ('PACKAGE','PROCEDURE','FUNCTION','TYPE')
+  if (ora_dict_obj_owner = UPPER('&&UT3_OWNER')
+    and ora_dict_obj_name = 'UT3_TRIGGER_ALIVE'
+    and ora_dict_obj_type = 'SYNONYM')
+  then
+    execute immediate 'begin ut_trigger_check.is_alive(); end;';
+  elsif   ( ora_dict_obj_type in ('PACKAGE','PROCEDURE','FUNCTION','TYPE')
         and ora_dict_obj_owner
           not in (
             'ANONYMOUS','APPQOSSYS','AUDSYS','DBSFWUSER','DBSNMP','DIP','GGSYS','GSMADMIN_INTERNAL',
             'GSMCATUSER','GSMUSER','ORACLE_OCM','OUTLN','REMOTE_SCHEDULER_AGENT','SYS','SYS$UMF',
             'SYSBACKUP','SYSDG','SYSKM','SYSRAC','SYSTEM','WMSYS','XDB','XS$NULL')
         and not (ora_dict_obj_type = 'TYPE' and ora_dict_obj_name like 'SYS\_PLSQL\_%' escape '\')
         )
-     or (ora_dict_obj_owner = UPPER('&&UT3_OWNER')
-         and ora_dict_obj_name = 'UT3_TRIGGER_ALIVE'
-         and ora_dict_obj_type = 'SYNONYM')
   then
     execute immediate 'begin ut_annotation_manager.trigger_obj_annotation_rebuild; end;';
   end if;

diff --git a/source/core/annotations/ut_trigger_check.pkb b/source/core/annotations/ut_trigger_check.pkb
@@ -29,7 +29,7 @@ create or replace package body ut_trigger_check is
 
   procedure is_alive is
   begin
-    if ora_dict_obj_owner = ut_utils.ut_owner and ora_dict_obj_name = gc_check_object_name and ora_dict_obj_type = 'SYNONYM' then
+    if ora_dict_obj_owner is not null and ora_dict_obj_name is not null and ora_dict_obj_type is not null then
       g_is_trigger_live := true;
     else
       g_is_trigger_live := false;

diff --git a/source/core/annotations/ut_trigger_check.pks b/source/core/annotations/ut_trigger_check.pks
@@ -24,7 +24,7 @@ create or replace package ut_trigger_check authid definer is
   function is_alive return boolean;
 
   /**
-  * If called from a DDL trigger when creating object gc_check_object_name, sts alive flag to true
+  * If called from a DDL trigger when creating object gc_check_object_name, sets alive flag to true
   * Otherwise sets alive flag to false.
   */
   procedure is_alive;

diff --git a/source/install.sql b/source/install.sql
@@ -31,7 +31,7 @@ prompt &&line_separator
 alter session set current_schema = &&ut3_owner;
 
 @@check_object_grants.sql
-@@check_sys_grants.sql
+@@check_sys_grants.sql "'CREATE TYPE','CREATE VIEW','CREATE SYNONYM','CREATE SEQUENCE','CREATE PROCEDURE','CREATE TABLE'"
 --set define off
 
 --dbms_output buffer cache table