Controller logging improvements

[matthchr]

What this PR does / why we need it:
We weren't consistent with what we were logging at what level. Some log level 0 messages were getting emitted every time through the reconcile loop, others were much less verbose. When writing a log statement, it was hard to determine where in the 0 to 10 spectrum to put your log without crawling all of the other log statements around you to rebuild your mental model of what levels we were using.

There is some relatively standard guidance from Kubernetes, but the problem is remembering it, especially given that they have 6 levels discussed there many of which are just described as "More details" which of course begs the question: "When do I choose "details" and when do I choose "more details""

The solution I came up with was to:

• Name some log levels and use the names. Hopefully the names are sorta descriptive.
• Provide documentation/examples that are specific to our use case (authoring a controller) on each named level for the kinds of logs that should go into that level.

I'm open to changing the log level names or even doing something else entirely, as long as it solves the fundamental problem of helping us keep our logs consistent.

I also set the log level in the container to 1 (I believe the default is 0) so that if we have issues escalated to us we have a bit more.

[matthchr]

Should we set this higher?

[theunrepentantgeek]

While we're in Alpha, yes - we kind of expect issues. Change it back to 1 when we hit Beta, maybe?

[matthchr]

I've put this up to 2 for now. That means we're not logging full API requests/responses (which we get at 3) but we're logging everything else.

Avoiding full requests/responses (except in our test logs for now) seems prudent given they could theoretically contain secrets at the moment since we don't do any pruning AND we don't put secrets into their own Kubernetes secrets object. We'll need to do something about that - I've got #1762 filed tracking that.

[codecov-commenter]

Codecov Report

Merging #1759 (2deb4c0) into master (a82afe8) will decrease coverage by 0.00%.
The diff coverage is 79.31%.

@@            Coverage Diff             @@
##           master    #1759      +/-   ##
==========================================
- Coverage   42.04%   42.04%   -0.01%     
==========================================
  Files         311      311              
  Lines       89943    89944       +1     
==========================================
- Hits        37821    37819       -2     
- Misses      48785    48788       +3     
  Partials     3337     3337              

Impacted Files	Coverage Δ
...ted/pkg/reconcilers/azure_deployment_reconciler.go	66.15% <77.77%> (-0.60%)	⬇️
hack/generated/controllers/generic_controller.go	78.37% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a82afe8...2deb4c0. Read the comment docs.

[theunrepentantgeek]

This is a really good idea.

I have a couple nitpicks on levels for some messages, but otherwise looks great.

[theunrepentantgeek]

If something stalls and never makes progress, this log line might be the key to spotting the situation.
Suggest restoring this, but at Verbose level.

[matthchr]

I removed this because we actually already logged this above on line 355 pretty much, so this line was duplicate. Actually the above line is better because it also includes the current Ready condition and hasChanged.

[theunrepentantgeek]

While we're in Alpha, yes - we kind of expect issues. Change it back to 1 when we hit Beta, maybe?