Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VCH deletion fails with CDE when the VCH has many container VMs #7020

Open
lgayatri opened this issue Jan 3, 2018 · 3 comments
Open

VCH deletion fails with CDE when the VCH has many container VMs #7020

lgayatri opened this issue Jan 3, 2018 · 3 comments
Labels
component/install kind/defect Behavior that is inconsistent with what's intended priority/p2 status/needs-estimation The issue needs to be estimated by the team team/lifecycle

Comments

@lgayatri
Copy link
Contributor

lgayatri commented Jan 3, 2018

Build: https://storage.cloud.google.com/vic-product-ova-releases/vic-v1.3.0-rc5-2996-f8cc7317.ova

Steps:

  1. Created 50 VCHs with 53 containers each.
  2. While the containers are running, I deleted all 50 VCHs with 20 minute timeout and --force option
  3. Delete failed with
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
2018/01/03 00:39:16 markDiskAsDeletable([CPBU_2TB_01] 32be64595b22a953e4576cd590dd6f5e14a533bfc0f6da70b94f16917eb4ee05/32be64595b22a953e4576cd590dd6f5e14a533bfc0f6da70b94f16917eb4ee05.vmdk): %!s(<nil>)
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="Failed to destroy VM \"VirtualMachine:vm-51037\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-52377\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-52792\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-51205\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-51937\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-52324\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-52061\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded"
time="2018-01-03T00:39:16-05:00" level=warning msg="Proceeding with delete of VCH due to --force"
time="2018-01-03T00:39:16-05:00" level=info msg="Removing image stores"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=info msg="Removing volume stores"
time="2018-01-03T00:39:16-05:00" level=info msg="Removing appliance VM network devices"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="Failed to get vm power status \"VirtualMachine:vm-50712\": Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded"
time="2018-01-03T00:39:16-05:00" level=info msg="Collecting e53fcd49-9375-46f0-9370-1da83c764012 vpxd.log"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="Failed to collect e53fcd49-9375-46f0-9370-1da83c764012 vpxd.log: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded"
time="2018-01-03T00:39:16-05:00" level=warning msg="No log data for e53fcd49-9375-46f0-9370-1da83c764012 vpxd.log"
time="2018-01-03T00:39:16-05:00" level=error msg="Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nPost https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/cmd/vic-machine/delete.(*Uninstall).Run.func2: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/cmd/vic-machine/delete.(*Uninstall).Run.func2: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg=--------------------
time="2018-01-03T00:39:16-05:00" level=error msg="vic-machine-linux delete failed: Delete timed out: use --timeout to add more time\n"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"

Operations were taking longer than 20 minutes to complete, hence the above error. Attached vi-machine.log.

vic-machine.zip

Few snippets from log:

Line 121128: Jan  3 2018 00:39:23.206-05:00 ERROR op=1374.1: Failed to collect e53fcd49-9375-46f0-9370-1da83c764012 vpxd.log: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded
...
Line 121171: Jan  3 2018 00:39:25.559-05:00 ERROR op=1236.1: Failed to destroy VM "VirtualMachine:vm-51654": ServerFaultCode: The method is disabled by 'VIC' then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded
...
Line 121258: Jan  3 2018 00:39:26.670-05:00 ERROR op=1365.1: vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded

@mdubya66 @cgtexmex - Not a blocker for 1.3 as retry helps delete. Can we document this?

@mdubya66
Copy link
Contributor

mdubya66 commented Jan 4, 2018

The workaround is to retry the delete. Adding kind/note.

@stuclem
Copy link
Contributor

stuclem commented Jan 9, 2018

Attempted release note:

  • Deleting multiple VCHs fails. #7020
    Deleting multiple VCHs at the same time fails with the error vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded.

    Workaround: Try the delete operation again.

@lgayatri is this OK?

@zjs zjs added the status/needs-estimation The issue needs to be estimated by the team label Jan 9, 2018
@stuclem
Copy link
Contributor

stuclem commented Jan 10, 2018

Thanks @AngieCris

@stuclem stuclem removed the impact/doc/note Requires creation of or changes to an official release note label Jan 10, 2018
@AngieCris AngieCris removed their assignment Jan 10, 2018
@zjs zjs changed the title VCH deletion fails with "vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux" VCH deletion fails with CDE when the VCH has many container VMs Apr 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/install kind/defect Behavior that is inconsistent with what's intended priority/p2 status/needs-estimation The issue needs to be estimated by the team team/lifecycle
Projects
None yet
Development

No branches or pull requests

5 participants