[dlp] fix: fix periodic builds timeout

[tmatsuo]

fixes #3375
fixes #3416
fixes #3417

• remove gcp-devrel-py-tools
• remove wrong usage of eventually_consistent.call
• only test if the operation has been started
• shorter timeout for polling
• correct use of pytest.mark.flaky

Amusingly, the usage of eventually_consistent.call in inspect_content_test.py is completely wrong.

The sample code is already polling with a timeout of 300 seconds by default. Then the test is wrapped with @eventually_consistent.call and @flaky, which means it would take more than 1 hour 40 mins (300 seconds * 10 * 2) in the worst case failure.

I made a change for datastore tests, for only checking if the operation has been started, instead of checking the operation results. I filed #3422 to investigate it further.

[tmatsuo]

The failure in py3.7 is interesting. If the API is too quick, it will fail.

Update:
I think I'm wrong. Now I'm adding retry.

[tmatsuo]

@kurtisvg Thanks! PTAL

[kurtisvg]

Does this not work directly on test_list_dlp_jobs?

[tmatsuo]

I doubt it. I got a feeling that pytest needs a plugin for decorator to work with pytest (think that @flaky doesn't work with pytest, you need to use @pytest.mark.flaky).

[tmatsuo]

Of course, we can use @pytest.mark.flaky, but I'd stick with @retry because of a semantic reason. But technically I can use @pytest.mark.flaky too.

[tmatsuo]

test_list_dlp_jobs_with_filter is still failing.

I can not repro locally at all. Maybe it's only failing when there are multiple builds.

I found out that create_dlp_job has an optional job_id parameter:
https://googleapis.dev/python/dlp/latest/gapic/v2/api.html

Maybe I can use that.

[tmatsuo]

Using uuid seems fix the issue. Maybe we don't need the retry at all.

[tmatsuo]

Wow jobs_test.py::test_list_dlp_jobs_with_filter is still failing, marking as flaky.

[tmatsuo]

With flaky, it's cumbersome to add delay. I think backoff looks good:
https://pypi.org/project/backoff/

I added backoff with max_time=60.

[tmatsuo]

I think I understand the reason of failure. test_list_dlp_jobs_with_filter has the following code:

   filter_string="state=RUNNING"

This means the list call is printing out the jobs which are still running. My first comment in #3420 (comment) was actually right.

I think I can change the filter with something else.

[tmatsuo]

Ha!

The error:

Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/dlp/risk_test.py", line 308, in test_k_map_estimate_analysis_single_field
    ["AGE"],
  File "/tmpfs/src/github/python-docs-samples/dlp/risk.py", line 661, in k_map_estimate_analysis
    subscription.result(timeout=timeout)
  File "/tmpfs/src/github/python-docs-samples/dlp/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/pubsub_v1/futures.py", line 105, in result
    raise err
  File "/tmpfs/src/github/python-docs-samples/dlp/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 71, in _wrap_callback_errors
    callback(message)
  File "/tmpfs/src/github/python-docs-samples/dlp/risk.py", line 575, in callback
    if message.attributes["DlpJobName"] == operation.name:
NameError: free variable 'operation' referenced before assignment in enclosing scope

[andrewsg]

It looks like you have a handle on this now; I didn't have any particular insight as it's been some years since I looked at this but let me know if you still need my input. Thank you!