Syslog Output for Chef Runs

A new blog post category was added to my blog for this post: Yak Shave. It was a small yak shave, but a table-flip inducing yak shave nonetheless.

UPDATE 6/3/2015: Per Lamont Granquist of Chef, As of Chef 12.4, the following works in client.rb without any other cookbooks needed (negating this whole blog post finally!):

log_location Chef::Log::Syslog.new("chef-client", ::Syslog::LOG_DAEMON)

THE FOLLOWING BLOG POST IS KEPT FOR HISTORICAL PURPOSES ONLY.

If you don’t want to read a pissy rant, don’t read any further (you can jump to the answer). If you’re going to comment to suggest Puppet, CFEngine, Ansible, or SaltStack, save your typing ;)

Begin

There you are with your configuration management tool. You say to yourself:

Self, surely there is a way to configure this damn-near-Linux-centric systems management tool to syslog its output so that, based on the common sane IT pattern of syslogging to a central server, all Chef run output can be stored in a centralized location and perhaps queried via Splunk or whatever other OSS tool. Because, you know, syslog has been around for 20+ years and is the de-facto means of logging information on UNIX/Linux hosts. And CFengine 2, dating back at least 15 years, is a CM tool that has offered such incredibly basic functionality. Surely the default of logging to stdout is just the absolute safe case that the product must default to, but there is definitely an option to turn on syslog as the log output location…

Knowing that this sort of thing would be defined as a “Chef Report Handler”, and that Chef has been alive for 4 years now, you start with Google: “chef syslog handler”

Handler – chef-syslog-handler

Bingo! The first search result is chef-syslog-handler, a “Chef handler to send syslog messages”.

You visit the link which takes you to rubygems.org, click on “Homepage” and get a 404 error from github.

Maybe the author renamed the repository? Viewing all of the author’s repositories shows that to be untrue. It’s simply gone. Returning to the rubygems page, you see “August 19, 2011”

Welp.

The Old Magically Worked Ticket

Returning to our Google results, we scroll down past all of the chef-syslog-handler results and end up at an old Opscode JIRA ticket.

Yes! This must be good news. The ticket is marked as Fixed!

17/Mar/11 5:49 PM
This is done... configure chef like this:

    log_location SyslogLogger.new("chef-client")

My follow-up comment to the ticket says it all:

I went to implement this today per snippet above.

It does not work with Chef 11.4.0

Looking through the mixlib-log 1.3.0+ versions on github, and looking at 10-stable, I don’t understand how it would have ever worked there either, but surely that’s just my ignorance.

WELP.

Revise Search

This is going nowhere past 2011, fast. Let’s feed Google something more generic: “chef syslog”

Cookbook – chef-client_syslog

Described as “Send chef-client log to syslog”, this too seemed promising.

And then I see that it does its own management of /etc/chef/client.rb via a template. As I said above, any sane person is handling via the chef-client cookbook.

SyslogLogger log_location

And finally, if you’re “lucky” (I forget how I got there), you end up at CHEF-2560 from August 11th 2011 which is still open!

Oh. It’s someone reporting, like I did, that the old Magically Worked Ticket’s solution does not work at all. It even includes code to fix it (unchecked), yet the ticket remains just dangling in limbo for the last 18 months.

Let’s follow the link to the pull request!

Here we find our old friend “pmorton” (the author of the old chef-syslog-handler gem in the first section of this blog post):

@guillermo – This is some great stuff. I seem to have located the ticket for this pull request (http://tickets.opscode.com/browse/CHEF-2560) but noticed two issues, the ticket has not been resolved and you are not part of the approved contributors list. I hope that you will take the time to get a CLA signed and sent to opscode. See http://wiki.opscode.com/display/chef/How+to+Contribute for more details.

The pull request is then closed out by Opscode due to lack of Contributor License Agreement. The original JIRA ticket CHEF-2560 remains open.

Fitna Cut Someone

This is where, per @jordansissel, I entered Hate-Driven Development mode.

I vowed to fix this once and for all, submit the code, and set aflame anyone at Opscode who rejected it.

Then I stumbled across one more thing.

Bleeding Edge chef-client Cookbook

UPDATE: THE BLOG POST IS KEPT FOR HISTORICAL PURPOSES ONLY. The solution here no longer works and I give up chasing this elusive dream.

Again, I don’t recall how I got there, but the current master branch of the chef-client community cookbook has a syslog example! That looks promising. Heh, cute, it references yet another JIRA ticket about syslog support, COOK-2326 (which links to almost everything listed above).

AND SO … if you are willing to make the leap to the heavily refactored latest chef-client cookbook that was released around June 11, 2013, you have a clean method of syslogging your output!

# our-chef-client-wrapper/files/default/syslog.rb

require 'rubygems'
require 'syslog-logger'
require 'syslog'

Logger::Syslog.class_eval do
  attr_accessor :sync, :formatter
end

log_location Logger::Syslog.new('chef-client', Syslog::LOG_DAEMON)

# our-chef-client-wrapper/recipes/whatever.rb

chef_gem 'syslog-logger'

# Drop off our chef-client customization file that
# allows for syslogging
cookbook_file '/etc/chef/client.d/syslog.rb' do
  source 'syslog.rb'
  mode 00644
  notifies :create, 'ruby_block[reload_client_config]', :immediately
end

include_recipe 'chef-client::config'

And this, my friends, is why you better know how to navigate the world of Ruby, Ruby gems, OSS, if you ever intend to have a remote chance with Chef. You will note that there was not a SINGLE THING in the experience above that was even close to being “sysadmin” friendly.

And the solution requires installing a gem at a critical point in what you want to be rock solid — the reporting of your 2000 machines’ chef runs. Behind an HTTP proxy? Hopefully you’ve run the chef-client cookbook already so that ENV['http_proxy'] is set so that the gem can fetch the right stuff.

You’d think Ruby didn’t have native syslog support in it’s standard library since version 1.8.6 or something.

Cause, you know… syslog is obscure. Who would ever want to use the logging mechanism used by the OS daemons since 1990… when running a daemon… that manages the OS and its daemons.