Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable RTC sync in chronyd. #142

Merged
merged 2 commits into from
Jan 10, 2019
Merged

Enable RTC sync in chronyd. #142

merged 2 commits into from
Jan 10, 2019

Conversation

agcooke
Copy link
Contributor

@agcooke agcooke commented Jan 9, 2019

Issue #, if available:
The chronyd service was running but not syncing the RTC clock.

This alarm was still firing:

ALERT ClockSyncBroken
  IF          node_timex_sync_status != 1
  FOR         5m
  LABELS      { severity="warning" }
  ANNOTATIONS {
    summary = "The clock is not being synced.",
    impact = "Random things are about to break for our users",
    detail = "Node: {{$labels.node}}",
  }

as described in #130

Description of changes:
After comparing the configuration between an Ubuntu AMI and the AL2 AMI I saw that the chrony.conf differerd.

AL2 was configured to not sync the RTC clock.

After making the changes it does sync and the alarm disappeared:

[ec2-user@ip-10-0-19-151 ~]$ sudo systemctl start chronyd
[ec2-user@ip-10-0-19-151 ~]$ ./adjtimex 
    mode:         0
-o  offset:       0 us
-f  freq.adjust:  2155554 (65536 = 1ppm)
    maxerror:     1098
    esterror:     564
    status:       0 ()
-p  timeconstant: 7
    precision:    1 us
    tolerance:    32768000
-t  tick:         9994 us
    time.tv_sec:  1547032282
    time.tv_usec: 176261
    return value: 0 (clock synchronized)

Links:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@mogren mogren requested a review from mattlandis January 9, 2019 17:33
Copy link
Member

@micahhausler micahhausler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! One small nit

install-worker.sh Outdated Show resolved Hide resolved
Copy link
Contributor

@mattlandis mattlandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes.

@micahhausler micahhausler merged commit 7f6c8cb into awslabs:master Jan 10, 2019
@errordeveloper
Copy link

Thanks for digging into this, we've noticed the problem a while ago, but it wasn't clear to me whether it's how node exporter implements the probe is inappropriate for all NTP daemons.

@agcooke
Copy link
Contributor Author

agcooke commented Jan 11, 2019

No problem. We have had lot's of head aches with our nodes. We got our Nodes stable now. But I am not an expert on getting Nodes healthy, thus some of the hand wavy pull requests I opened here, I just wanted them to work.

Now you guys can take a look into fluxcd/flux#1639 ;)

We love Flux by the way!

@errordeveloper
Copy link

errordeveloper commented Jan 11, 2019 via email

@agcooke agcooke mentioned this pull request Jan 11, 2019
@Kausheel Kausheel mentioned this pull request Dec 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants