Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rsync option #254

Merged
merged 1 commit into from
Oct 3, 2014
Merged

Rsync option #254

merged 1 commit into from
Oct 3, 2014

Conversation

danzilio
Copy link
Contributor

We're using librarian-puppet in a large-scale environment with lots of change and we've run into issues with librarian-puppet's rather liberal convergence strategy. Essentially we're running into an issue where a librarian-puppet update command will result in a Puppet master process losing it's CWD because of the install_path.rmtree call. This results in a catalog compilation failure. The install_path.rmtree method gets called regardless of whether there are changes to the module in install_path. This PR allows for the option to use rsync to converge the directories, which is ultimately a much more conservative convergence strategy. This may not be the most idiomatic implementation. I'm open to suggestions on how to improve it.

@danzilio
Copy link
Contributor Author

I'll need to add some tests and documentation as well.

@carlossg
Copy link
Collaborator

Would your problem be solved if instead of rmtree the destdir it deletes destdir/* ?

The code looks good, would definitely need some docs and tests, it could even be the default and remove all the custom cp logic, permissions,...
It would be nice to have tests that reproduce your issue to ensure it doesn't happen again.

@njam
Copy link
Contributor

njam commented Sep 18, 2014

Cool cool cool.
We've implemented such functionality within our puppet master code.
But obviously it makes much more sense in librarian-puppet!

I would even vote to make this default, because when you create a puppet master's module-directory with librarian-puppet sooner or later the problem's gonna arise that puppet reads files that are being deleted at the same time.

cc @tomaszdurka @ppp0 @kris-lab

@danzilio
Copy link
Contributor Author

The only reason I would hesitate about making this the default behavior is that we would not be able to control the dependency on rsync.

@carlossg I haven't tested the destdir/* scenario but I imagine it wouldn't solve the issue since the Puppet master process is likely reading from destdir/manifests at the time rmtree is called.

@njam thanks for chiming in. I was wondering if I was the only one seeing this behavior!

I'll add some tests and docs today.

@danzilio
Copy link
Contributor Author

I just pushed a cucumber scenario that captures this, but it feels kind of hacky. Could I get some input on the Updating a module with the rsync configuration scenario in features/updates.feature? If this approach is sound, I'll add some more scenarios.

@carlossg
Copy link
Collaborator

Looks good to me, doesn't the same problem happen when running librarian-puppet install after the first time?

@danzilio
Copy link
Contributor Author

It does, I'll add a couple more tests and docs soon. I've been swamped with PuppetConf stuff and I haven't had a chance to touch this for the past few days.

@danzilio
Copy link
Contributor Author

danzilio commented Oct 1, 2014

What do you guys think? Is this code ready to go?

@danzilio
Copy link
Contributor Author

danzilio commented Oct 1, 2014

Actually, I've found one issue with this. I'm going to add another test case and fix it.

@carlossg
Copy link
Collaborator

carlossg commented Oct 2, 2014

Can you squash the commits into one for merging?

Need to use the parent of dest for rsync

Reverting gemfile

Adding a test. Going to check with maintainers to see if this is a valid testing strategy.

Adding ctime to the mix to make sure that the file isn't just assigned the same inode

Adding tests for the rsync option using the install command

Only compare ctime if inodes match

Adding test case to make sure update works properly with git
@danzilio
Copy link
Contributor Author

danzilio commented Oct 2, 2014

Squashed :)

@carlossg
Copy link
Collaborator

carlossg commented Oct 3, 2014

Thanks!

carlossg added a commit that referenced this pull request Oct 3, 2014
@carlossg carlossg merged commit 2be40b9 into rodjek:master Oct 3, 2014
@carlossg
Copy link
Collaborator

carlossg commented Oct 3, 2014

Seems there are some timing issues ?
https://travis-ci.org/rodjek/librarian-puppet/jobs/36945009#L188

@danzilio
Copy link
Contributor Author

danzilio commented Oct 3, 2014

Strange. I'll dig into it!

@danzilio
Copy link
Contributor Author

danzilio commented Oct 3, 2014

Do you want to revert for now?

@carlossg
Copy link
Collaborator

carlossg commented Oct 3, 2014

No if it doesn't take too long to fix. I've tried with 7506ced but seems it didn't fix it

@danzilio
Copy link
Contributor Author

danzilio commented Oct 3, 2014

I think it may be the logic in my test. Hopefully I'll have a commit for you in a few hours.

@danzilio
Copy link
Contributor Author

danzilio commented Oct 7, 2014

Sorry, I've had a heck of a time replicating this.

@carlossg
Copy link
Collaborator

seems it only happens in the 1.x branch

carlossg pushed a commit that referenced this pull request Oct 13, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants