Deprecated AI Platform Operators and Runtimes in Example DAG

[vuppalli]

Apache Airflow version: 1.10.10

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

• Cloud provider or hardware configuration: GCP
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

The file, /airflow/providers/google/cloud/example_dags/example_mlengine.py, uses deprecated operators and a soon to be deprecated AI Platform runtime version.

• MLEngineManageModelOperator is used to create and get a model. (https://airflow.readthedocs.io/en/latest/_api/airflow/providers/google/cloud/operators/mlengine/index.html#airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator)

create_model = MLEngineManageModelOperator(
        task_id="create-model", 
        project_id=PROJECT_ID, 
        operation='create',
        model={
            "name": MODEL_NAME,
        },
)

get_model = MLEngineManageModelOperator(
        task_id="get-model",
        project_id=PROJECT_ID,
        operation="get",
        model={
            "name": MODEL_NAME,
        }
)

• Many operators use a runtime version of 1.14 which will be deprecated next month. (https://cloud.google.com/ai-platform/training/docs/runtime-version-list#1.14)

create_version = MLEngineCreateVersionOperator(
        task_id="create-version", 
        project_id=PROJECT_ID, 
        model_name=MODEL_NAME,
        version={
            "name": "v1",
            "description": "First-version",
            "deployment_uri": '{}/keras_export/'.format(JOB_DIR),
            "runtime_version": "1.14",
            "machineType": "mls1-c1-m2",
            "framework": "TENSORFLOW",
            "pythonVersion": "3.5"
        }
)

What you expected to happen:

The file, /airflow/providers/google/cloud/example_dags/example_mlengine.py, should be using new operators and a more recent AI Platform runtime version.

• To create a model: MLEngineCreateModelOperator
• To get a model: MLEngineGetModelOperator
• All operators should use a runtime version of 1.15 or 2.1.

How to reproduce it:

Anything else we need to know:
I would be happy to take this issue!

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

I am very happy that you want to join this project. I assigned you to this ticket. I am the maintainer of this integration, so if you have questions, ask I look forward to your contribution.

I also recommend reading our contribution guide.
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst

I would like to pay particular attention to one fragment.

If you create pull-request, you don't have to create an issue first, but if you want, you can do it. Creating an issue will allow you to collect feedback or share plans with other people.

[vuppalli]

Thanks so much! I have a couple of quick questions: when running the DAG, I noticed that I do not have access to the GCP resources used in the file. Would it be possible to get added to the project so that I can run the DAG without making any changes successfully? Or, is there a place where I can access all of these resources for myself? Additionally, is there an official test file to confirm that the DAG works?

[mik-laj]

Unfortunately, the community has not developed procedures for storing files necessary for testing. We don't run these tests automatically on CI yet. Each team that works on integrations must provide these files on their own. This should not be difficult based on official guides for this service.
https://cloud.google.com/ai-platform/training/docs/

I have the following environment variables configured to run these tests.

GCP_PROJECT_ID=polidea-airflow

GCP_MLENGINE_BUCKET_NAME_EPHEMERAL=polidea-airflow-tests-38
GCP_MLENGINE_BUCKET_NAME_PERSISTENT=test-airflow-mlengine-persistent

GCP_MLENGINE_MODEL_NAME=airflow_test_ci_model_name_19837

GCP_MLENGINE_PREDICTION_INPUT=gs://test-airflow-mlengine-persistent/prediction_input.json
GCP_MLENGINE_TRAINER_URI=gs://test-airflow-mlengine-persistent/trainer-0.0.0.tar.gz

GCP_MLENGINE_DATAFLOW_STAGING=gs://polidea-airflow-tests-38/staging/
GCP_MLENGINE_DATAFLOW_TMP=gs://polidea-airflow-tests-38/tmp/
GCP_MLENGINE_JOB_DIR=gs://polidea-airflow-tests-38/job-dir
GCP_MLENGINE_PREDICTION_OUTPUT=gs://polidea-airflow-tests-38/prediction_output/
GCP_MLENGINE_SAVED_MODEL_PATH=gs://polidea-airflow-tests-38/job-dir/keras_export/

I prepared copies of the necessary files for you and made them available in a public bucket. Please make a copy of this bucket if you want to work on this integration, because this bucket can be deleted at any time.

gs://airflow-polidea-googl-system-tests-resources-public

I hope this information will help you.

[vuppalli]

Thank you for the information! I created a PR for this issue here: #9727.