[docs] voyager-1.7.2 changes

docs/content/preview/yugabyte-voyager/known-issues/mysql.md

@@ -29,6 +29,10 @@ Review limitations and implement suggested workarounds to successfully migrate d
 - [Exporting text type columns with default value](#exporting-text-type-columns-with-default-value)
 - [json_valid() does not exist in PostgreSQL/YugabyteDB](#json-valid-does-not-exist-in-postgresql-yugabytedb)
 - [json_value() does not exist in PostgreSQL/YugabyteDB](#json-value-does-not-exist-in-postgresql-yugabytedb)
+- [Unnecessary DDLs for RANGE COLUMN PARTITIONED tables](#unnecessary-ddls-for-range-column-partitioned-tables)
+- [DOUBLE UNSIGNED and FLOAT UNSIGNED datatypes are not supported](#double-unsigned-and-float-unsigned-datatypes-are-not-supported)
+- [Foreign key referenced column cannot be a subset of a unique/primary key](#foreign-key-referenced-column-cannot-be-a-subset-of-a-unique-primary-key)
+- [Multiple indexes on the same column of a table errors during import](#multiple-indexes-on-the-same-column-of-a-table-errors-during-import)
 
 ### Approaching MAX/MIN double precision values are not imported
 
@@ -580,3 +584,165 @@ Suggested change to the schema is as follows:
 ```sql
 json_extract_path_text(key_value_pair_variable::json,'key');
 ```
+
+---
+
+### Unnecessary DDLs for RANGE COLUMN PARTITIONED tables
+
+**GitHub**: [Issue #511](https://github.com/yugabyte/yb-voyager/issues/511)
+
+**Description**: If you have a schema which contains RANGE COLUMN PARTITIONED tables in MYSQL, some extra indexes are created during migration which are unnecessary and might result in an import error.
+
+**Workaround**: Remove the unnecessary DDLs from `PARTITION_INDEXES_partition.sql` file from the export-dir/schema/partitions sub-directory for the corresponding tables.
+
+**Example**
+
+An example schema on the source database is as follows:
+
+```sql
+CREATE TABLE range_columns_partition_test (
+a INT,
+b INT
+)
+PARTITION BY RANGE COLUMNS(a, b) (
+PARTITION p0 VALUES LESS THAN (5, 5),
+PARTITION p1 VALUES LESS THAN (MAXVALUE, MAXVALUE)
+);
+```
+
+An example of the exported unnecessary index is as follows:
+
+```sql
+-- Create indexes on each partition of table range_columns_partition_test
+CREATE INDEX range_columns_partition_test_p1_b ON range_columns_partition_test_p1 (b);
+```
+
+Suggested change is to remove the unnecessary index.
+
+---
+
+### DOUBLE UNSIGNED and FLOAT UNSIGNED datatypes are not supported
+
+**GitHub**: [Issue #1607](https://github.com/yugabyte/yb-voyager/issues/1607)
+
+**Description**: If the schema has a table with a DOUBLE UNSIGNED or FLOAT UNSIGNED column, those datatypes  are not converted by Voyager to a YugabyteDB relevant syntax, and thus get errored during import. These datatypes are deprecated as of [MySQL 8.0.17](https://dev.mysql.com/doc/refman/8.0/en/numeric-type-syntax.html).
+
+**Workaround**: Manually change the exported schema to the closest YugabyteDB supported syntax.
+
+**Example**:
+
+An example schema on the source database is as follows:
+
+```sql
+CREATE TABLE test(id int,
+                  n double unsigned,
+                  f float unsigned
+);
+```
+
+An example of the exported table is as follows:
+
+```sql
+CREATE TABLE test (
+        id bigint,
+        n DOUBLE UNSIGNED,
+        f FLOAT UNSIGNED
+);
+```
+
+Suggested change to the schema is as follows:
+
+Change to YugabyteDB compatible syntax closest to the respective datatypes.
+
+```sql
+CREATE TABLE test (
+        id bigint,
+        n DOUBLE PRECISION CHECK (n >= 0),
+        f REAL CHECK (f >= 0)
+);
+```
+
+---
+
+### Foreign key referenced column cannot be a subset of a unique/primary key
+
+**GitHub**: [Issue #1608](https://github.com/yugabyte/yb-voyager/issues/1608)
+
+**Description**: In MySQL, you can reference a column which is part of a unique key or a primary key for a foreign key. This is not allowed in YugabyteDB or PostgreSQL.
+
+**Workaround**: The referenced columns will need to have individual Unique / Primary keys.
+
+**Example**:
+
+An example schema on the source database (allowed in MySQL) is as follows:
+
+```sql
+create table k(id int,id2 int,UNIQUE KEY `uk_k` (`id`,`id2`));
+
+create table h(id int,CONSTRAINT `fk_h` FOREIGN KEY (`id`) REFERENCES `k` (`id`));
+```
+
+An example of the exported schema is as follows:
+
+```sql
+CREATE TABLE h (
+        id bigint
+) ;
+CREATE TABLE k (
+        id bigint,
+        id2 bigint
+) ;
+ALTER TABLE k ADD UNIQUE (id,id2);
+ALTER TABLE h ADD CONSTRAINT fk_h FOREIGN KEY (id) REFERENCES k(id) MATCH SIMPLE ON DELETE NO ACTION ON UPDATE NO ACTION;
+```
+
+Error during import is as follows:
+
+```output
+ERROR:  there is no unique constraint matching given keys for referenced table "k"
+```
+
+Suggested change to the schema is as follows:
+
+Add individual unique/primary keys to the referenced column as follows:
+
+```sql
+ALTER TABLE k ADD UNIQUE (id);
+```
+
+---
+
+### Multiple indexes on the same column of a table errors during import
+
+**GitHub**: [Issue #1609](https://github.com/yugabyte/yb-voyager/issues/1609)
+
+**Description**: Voyager renames the indexes with a set pattern `table_name_column_name` during export. So having multiple indexes on the same column of a table will have the same name and will error out during import.
+
+**Workarounds**:
+
+1. Rename the duplicate index after the schema export.
+1. Drop the duplicate index on the source.
+
+**Example**
+
+An example schema on the source database is as follows:
+
+```sql
+CREATE TABLE test(id int, k int);
+CREATE INDEX index1 on test(k);
+CREATE INDEX index2 on test(k);
+```
+
+An example of the exported indexes are as follows:
+
+```sql
+CREATE INDEX test_k ON test (k);
+CREATE INDEX test_k ON test (k);
+```
+
+Suggested changes:
+
+1. Rename one of the indexes (for example, test_k2).
+1. Drop index1 or index2 on the source before exporting.
+
+---

docs/content/preview/yugabyte-voyager/migrate/assess-migration.md

@@ -13,7 +13,7 @@ badges: tp
 type: docs
 ---
 
-The Voyager Migration Assessment feature is specifically designed to optimize the database migration process from various source databases, currently supporting PostgreSQL to YugabyteDB. Voyager conducts a detailed analysis of the source database by capturing essential metadata and metrics. It generates a comprehensive assessment report that recommends effective migration strategies, and provides key insights on ideal cluster configurations for optimal performance with YugabyteDB.
+The Voyager Migration Assessment feature is specifically designed to optimize the database migration process from various source databases, currently supporting PostgreSQL and Oracle to YugabyteDB. Voyager conducts a detailed analysis of the source database by capturing essential metadata and metrics. It generates a comprehensive assessment report that recommends effective migration strategies, and provides key insights on ideal cluster configurations for optimal performance with YugabyteDB.
 
 ## Overview
 
@@ -33,7 +33,7 @@ When you run an assessment, Voyager collects metadata or metrics from the source
 The recommendations are based on testing using a [RF3](../../../architecture/docdb-replication/replication/#replication-factor) YugabyteDB cluster on instance types with 4GiB memory per core and running v2024.1.
 {{< /warning >}}
 
-Note that for cases where it is not possible to provide database access to the client machine running Voyager, you can gather the metadata from the source database using plain bash/psql scripts provided with Voyager, and then use Voyager to analyze the metadata directly.
+Note that if providing database access to the client machine running Voyager is not possible, you can gather metadata from the source database using the provided bash scripts and then use Voyager to assess the migration.
 
 The following table describes the type of data that is collected during a migration assessment.
 
@@ -46,9 +46,9 @@ The following table describes the type of data that is collected during a migrat
 | Performance metrics | Optional | Voyager captures performance metrics from the database (IOPS) for rightsizing the target environment. |
 | Server or database credentials | No | No server or database credentials are collected. |
 
-### Sample Migration Assessment report
+### PostgreSQL Sample Migration Assessment report
 
-A sample Migration Assessment report is as follows:
+A sample Migration Assessment report for PostgreSQL is as follows:
 
 ![Migration report](/images/migrate/assess-migration.jpg)
 
@@ -57,28 +57,90 @@ A sample Migration Assessment report is as follows:
 1. [Install yb-voyager](../../install-yb-voyager/).
 1. Prepare the source database.
 
-    1. Create a new user, `ybvoyager` as follows:
+    {{< tabpane text=true >}}
 
-        ```sql
-        CREATE USER ybvoyager PASSWORD 'password';
-        ```
+      {{% tab header="PostgreSQL" %}}
 
-    1. Grant necessary permissions to the `ybvoyager` user.
+1. Create a new user, `ybvoyager` as follows:
 
-        ```sql
-        /* Switch to the database that you want to migrate.*/
-        \c <database_name>
+    ```sql
+    CREATE USER ybvoyager PASSWORD 'password';
+    ```
 
-        /* Grant the USAGE permission to the ybvoyager user on all schemas of the database.*/
+1. Grant necessary permissions to the `ybvoyager` user.
 
-        SELECT 'GRANT USAGE ON SCHEMA ' || schema_name || ' TO ybvoyager;' FROM information_schema.schemata; \gexec
+    ```sql
+    /* Switch to the database that you want to migrate.*/
+    \c <database_name>
 
-        /* The above SELECT statement generates a list of GRANT USAGE statements which are then executed by psql because of the \gexec switch. The \gexec switch works for PostgreSQL v9.6 and later. For older versions, you'll have to manually execute the GRANT USAGE ON SCHEMA schema_name TO ybvoyager statement, for each schema in the source PostgreSQL database. */
+    /* Grant the USAGE permission to the ybvoyager user on all schemas of the database.*/
 
-        /* Grant SELECT permission on all the tables. */
+    SELECT 'GRANT USAGE ON SCHEMA ' || schema_name || ' TO ybvoyager;' FROM information_schema.schemata; \gexec
 
-        SELECT 'GRANT SELECT ON ALL TABLES IN SCHEMA ' || schema_name || ' TO ybvoyager;' FROM information_schema.schemata; \gexec
-        ```
+    /* The above SELECT statement generates a list of GRANT USAGE statements which are then executed by psql because of the \gexec switch. The \gexec switch works for PostgreSQL v9.6 and later. For older versions, you'll have to manually execute the GRANT USAGE ON SCHEMA schema_name TO ybvoyager statement, for each schema in the source PostgreSQL database. */
+
+    /* Grant SELECT permission on all the tables. */
+
+    SELECT 'GRANT SELECT ON ALL TABLES IN SCHEMA ' || schema_name || ' TO ybvoyager;' FROM information_schema.schemata; \gexec
+    ```
+
+    {{% /tab %}}
+
+    {{% tab header="Oracle" %}}
+
+1. Create a role that has the privileges as listed in the following table:
+
+   | Permission | Object type in the source schema |
+   | :--------- | :---------------------------------- |
+   | `SELECT` | VIEW, SEQUENCE, TABLE PARTITION, TABLE, SYNONYM, MATERIALIZED VIEW |
+   | `EXECUTE` | TYPE |
+
+   Change the `<SCHEMA_NAME>` appropriately in the following snippets, and run the following steps as a privileged user.
+
+   ```sql
+   CREATE ROLE <SCHEMA_NAME>_reader_role;
+
+   BEGIN
+       FOR R IN (SELECT owner, object_name FROM all_objects WHERE owner=UPPER('<SCHEMA_NAME>') and object_type in ('VIEW','SEQUENCE','TABLE PARTITION','SYNONYM','MATERIALIZED VIEW'))
+       LOOP
+          EXECUTE IMMEDIATE 'grant select on '||R.owner||'."'||R.object_name||'" to <SCHEMA_NAME>_reader_role';
+       END LOOP;
+   END;
+   /
+
+   BEGIN
+       FOR R IN (SELECT owner, object_name FROM all_objects WHERE owner=UPPER('<SCHEMA_NAME>') and object_type ='TABLE' MINUS SELECT owner, table_name from all_nested_tables where owner = UPPER('<SCHEMA_NAME>'))
+       LOOP
+          EXECUTE IMMEDIATE 'grant select on '||R.owner||'."'||R.object_name||'" to  <SCHEMA_NAME>_reader_role';
+       END LOOP;
+   END;
+   /
+
+   BEGIN
+       FOR R IN (SELECT owner, object_name FROM all_objects WHERE owner=UPPER('<SCHEMA_NAME>') and object_type = 'TYPE')
+       LOOP
+          EXECUTE IMMEDIATE 'grant execute on '||R.owner||'."'||R.object_name||'" to <SCHEMA_NAME>_reader_role';
+       END LOOP;
+   END;
+   /
+
+   GRANT SELECT_CATALOG_ROLE TO <SCHEMA_NAME>_reader_role;
+   GRANT SELECT ANY DICTIONARY TO <SCHEMA_NAME>_reader_role;
+   GRANT SELECT ON SYS.ARGUMENT$ TO <SCHEMA_NAME>_reader_role;
+
+   ```
+
+1. Create a user `ybvoyager` and grant `CONNECT` and `<SCHEMA_NAME>_reader_role` to the user:
+
+   ```sql
+   CREATE USER ybvoyager IDENTIFIED BY password;
+   GRANT CONNECT TO ybvoyager;
+   GRANT <SCHEMA_NAME>_reader_role TO ybvoyager;
+   ```
+
+    {{% /tab %}}
+
+{{< /tabpane >}}
 
 1. Assess migration - Voyager supports two primary modes for conducting migration assessments, depending on your access to the source database as follows:
 
@@ -91,9 +153,9 @@ A sample Migration Assessment report is as follows:
         --source-db-schema schema1,schema2 --export-dir /path/to/export/dir
         ```
 
-    1. **Without source database connectivity**: In situations where direct access to the source database is restricted, there is an alternative approach. Voyager includes packages with scripts present at `/etc/yb-voyager/gather-assessment-metadata/postgresql`. You can perform the following steps with these scripts.
+    1. **Without source database connectivity** (only PostgreSQL): In situations where direct access to the source database is restricted, there is an alternative approach. Voyager includes packages with scripts for PostgreSQL at `/etc/yb-voyager/gather-assessment-metadata`. You can perform the following steps with these scripts:
 
-        1. Copy the scripts to a machine which has access to the source database.
+        1. On a machine which has access to the source database, copy the scripts and install dependencies psql, and pg_dump version 14 or later. Alternatively, you can install yb-voyager on the machine to automatically get the dependencies.
         1. Run the `yb-voyager-pg-gather-assessment-metadata.sh` script by providing the source connection string, the schema names, path to a directory where metadata will be saved, and an optional argument of an interval to capture the IOPS metadata of the source (in seconds with a default value of 120). For example,
 
             ```sh
@@ -115,7 +177,7 @@ For the most accurate migration assessment, the source database must be actively
 
 1. Create a target YugabyteDB cluster as follows:
 
-    1. Create a cluster based on the sizing recommendations in the assessment report.
+    1. Create a cluster in [Enhanced Postgres Compatibility Mode](/preview/releases/ybdb-releases/v2024.1/#highlights) based on the sizing recommendations in the assessment report. For a universe in YugabyteDB Anywhere, [enable the compatibility mode](/preview/releases/yba-releases/v2024.1/#highlights) by setting some flags on the universe.
     1. Create a database with colocation set to TRUE.
 
         ```sql

docs/content/preview/yugabyte-voyager/migrate/bulk-data-load.md

@@ -108,10 +108,17 @@ To import an updated version of the same file (that is, having the same file nam
 yb-voyager import data file --data-dir /dir/data-dir --file-table-map 'orders.csv:orders' ...
 ```
 
-After new rows are added to `orders.csv`, use the following command to load them with `--start-clean`:
+After new rows are added to `orders.csv`, use the following command to load them with the flags `--start-clean` and `--enable-upsert`.
+
+{{<warning title="Note">}}
+Ensure that tables on the target YugabyteDB database do not have secondary indexes. If a table has secondary indexes, using `--enable-upsert true` may lead to corruption of the indexes.
+{{</warning>}}
 
 ```sh
-yb-voyager import data file --data-dir /dir/data-dir --file-table-map 'orders.csv:orders' --start-clean ...`
+yb-voyager import data file --data-dir /dir/data-dir \
+        --file-table-map 'orders.csv:orders' \
+        --start-clean true \
+        --enable-upsert true
 ```
 
 For details about the argument, refer to the [arguments table](../../reference/bulk-data-load/import-data-file/#arguments).