stop expensive osx arm64 builds on every diff (facebookresearch#2835)

Summary:
Pull Request resolved: facebookresearch#2835

I don't think this provides much extra signal on top of Linux arm64, and we will still have the nightly. It's by far the most expensive to run, so let's save the $$$ and the planet.

Reviewed By: mlomeli1

Differential Revision: D45389825

fbshipit-source-id: 63fa6b37f3f7505f118c75f03605c065f1ad51f1

.circleci/config.yml

@@ -268,11 +268,6 @@ workflows:
       - build_conda:
           name: Windows x86_64 (conda)
           exec: windows-x86_64-cpu
-      - build_conda:
-          name: OSX arm64 (conda)
-          exec: macosx-arm64-cpu
-          requires:
-            - Linux arm64 (conda)
       - build_conda:
           name: Linux arm64 (conda)
           exec: linux-arm64-cpu