Posted by & filed under 'nix Tools, Pulp.

At work we’re looking to do patch management on our servers in a more logical and steady way. Right now we are going from e-mail to testing to running commands manually…it’s a mess, and because we are just now starting to get the hands we need, it goes sometimes longer than it should.

I’ve looked at some solutions and Pulp seems to have everything we want, which is a well-documented/supported subscription management service that lets us move through the steps of updating and tracking updates easily and sensibly.

I’m working on it in bits and pieces, recording steps along the way. Here is my shelr cap of a quick pulp install. It runs through some hiccups (in the middle there is a long pause you’ll want to speed past — I researched the problem between python-rhsm and subscription-manager), but you can see it’s otherwise pretty easy.

More to come as I work on it!

Posted by & filed under Instructions.

I am toying around with grive, a third party Google Drive client.  It’s pretty cool, but the instructions are based on an Ubuntu install.  I had an interesting time setting it up on CentOS 6.2, so I decided to share what I learned.

First, get EPEL:

rpm -ivh epel-release-6-7.noarch.rpm

Then set up Cmake. You can install from package but the yum package isn’t up to date enough, so to source we go!

mkdir /opt/src/cmake
cd /opt/src/cmake
curl >

extract to subdirectory. In my case I did it in /opt/src, so I run

/opt/cmake/cmake- /usr/bin/cmake

Now we’re going to install the packages we need:

sudo yum install -y libcurl-devel libgcrypt-devel boost-devel boost-program-options boost boost-filesystem binutils-devel libexpat-devel libgcrypt-devel json-c-devel git

Now time for grive:

cd /opt/src
git clone git://
cd /opt/src/grive
/opt/cmake/cmake- .
cd grive


When I ran it, I had this error:

[ 97%] Built target grive
make[2]: *** No rule to make target /usr/lib/lib/', needed bygrive/grive'. Stop.
make[1]: *** [grive/CMakeFiles/grive_executable.dir/all] Error 2
make: *** [all] Error 2

The problem is that usr/lib/lib doesn’t exist! So then edit that so there is only /usr/lib/ in these files:


Posted by & filed under Troubleshooting.

Repeat after me:  When a problem becomes very strange, I will always check the basics.

Not that this mantra prevents me from routinely forgetting this in the middle of some crazy problem. Too often what happens is I’m trying to fix a problem, iterating over different tests and hypothesis, until  the results stop making sense.  How can this configuration work here perfectly, but not on the similar machine  at all?  How do I have visibility to this part of my network but not this part?  Why does it show up intermittently? Just as I’ve exhausted every possibility, my mantra finally comes to my weary and exhausted brain: “Test your assumptions.”

Assumptions often get skipped.  “Oh that is definitely working!” If you’re troubleshooting in a group it’s even more likely. “Is this working?” “Of course it is.”  It can be socially risky to ask about certain things.  For instance, when I did app support, you’d swear that everyone’s firewall was always configured incredibly.  If you even asked about a firewall you’d get an earful.

The funny thing is that the likelihood of a problem with software resulting from a misconfigured firewall was inversely propportional to how fiercely someone denied it.  Often the real task was getting the customer to actually look.  When they did, very commonly you’d get a sheepish “Oh yeah, I guess it was” and everything would work.  Sometimes people would get so stuck in their assumptions that they do anything to avoid it.  They’d yell, question your competence, demand to speak to someone else.  Imagine what it’s like to make all this noise and then your problem is the one you demanded was not the problem, could not be the problem.

And it happens all the time! It happens to me.

In tech, we value competence highly.  We should.  Our toxic addiction to confusing rightness with competence is a blind-spot.  Knowing the answer to every question, and having everything done perfectly creates some of the weirdest working environments I’ve ever seen. It’s taken problems that should take 5 minutes to fix take 5 days, all because we are culturally trained in tech that if we are competent, we don’t have to question our assumptions.

The problem with that lies in the fact that though we work with computers, we are not computers; we are beautiful, fallible little engines and we should embrace that. When we don’t, our assumptions turn our inevitable errors into damaging blows against us. We need to make assumptions to move forward –we can’t give deep thought to everything in front of us– but making assumptions isn’t the crime.  The crime is not circling back when there could be a problem.  Normally the cost is so minimal to do this that it’s worth it to look. The cost of not doing so can be really high.

I’m having a network issue.  Quick, before I do anything else: is the NIC card working on this machine?  This is the type of thing we often assume is true, and don’t check until we’ve exhausted the more high-level options, finding only strange, perplexing results from our tests.

By often, of course, I mean this past weekend.  By we, I mean me.  The machine was in our DMZ and not in any of our monitoring systems yet, so I didn’t get any warning signs like I’d expect to.

My solution of course it to attempt to automate checks.  I’d like to be able to, when having an issue, run a quick script that checks very common thing and then gives me quick feedback on the items in question.  If I can make assumptions even more painless to test, I’m more willing to do them, and it’s easier to convince someone with heavy assumptions of their own.  “This will take a minute and answer some basic questions, just in case.”

How do you circumvent or avoid assumptions in your troubleshooting?

Posted by & filed under Essays.

I’m on a mission to collect servers and try different hosting schemes.

I’ve had a Dreamhost account forever.  Sometimes Dreamhost is flaky –it’s down, a file just goes missing and there’s no backup — but it does have a log of goodies and I have a ton of space from being on their so long.  I’ve got a reasonably priced VPN that I’ve used to store files and host IRC and run Ventrilo servers.  I won’t rant and rave, but I’m not going to complain.

I’ve got a Linode server.  Way less goodies, but that’s fine.  It gets out of your way and is really really stable.  Also, pretty damn cheap! $20/month for a rock solid space you can run any distro you want on is very nice.  They do have scripts that set up things for you, but I have my Dreamhost account when I need goodies.

Here is where I seek to add more and more servers to the pile.

I am going to add a server from Amazon’s free usage tier. EC2 is pretty amazing.

Lastly, I’m going to dust off an old mini-mac and put Linux on it at my house.

With this, I’ve got a nice distributed web for playing around and testing things on.  Here’s what I’m running or looking to run across this distributed network of computers:

  • collectd
  • graphite
  • logstash
  • rundeck
  • puppet

This is on top of all the apps and services I’m working on and will detail here!


Posted by & filed under Videos.

Discovered via HighScalability is this great video from the ops team at Etsy.  I think it is a really great view to how a startup evolves its practices.  Of note is that painful middle when the company is trying to scale well with its own success.  Definitely worth spinning up and listening to/watching.

Posted by & filed under 'nix Tools.

When I find something awesome, I go and I tell everyone I can about it.  No, I don’t mean awk, I mean ack.  You know, like ack-grep? Ack, like Bill the Cat? No?

Well, I know I use the venerable grep in the title of this very blog, but ack is a tool that I’m using more and more for my everyday work.  It’s grep with all the nooks and crannies filled or cleaned as needed.  What makes ack so great is not that it’s basic functionality is that much different from grep; at heart, they are both filtering tools.  Grep is insanely powerful, but there is so much that you have to ask it to do, every time.  Whenever I use grep, I need to think harder than I want to about what to hand it.  Common annoyances turn into a million and one wrapper scripts.

Because seeing is believing, let’s take a common scenario for me.  I’ve got to grep in a CVS directory for information but I don’t want any CVS-related files.  I do something like:

With ack, I do:

I haven’t even started to talk about the slick color formatting of the output (which you can of course turn off with –nocolor, but why?).  You can customize default behaviors (including additional files to exclude from searches) with an ~/.ackrc file.  When I first ran into , I thought to myself, “What hubris!”  I was completely, utterly wrong. Ack is better than grep in every way, a perfect extension of grep’s basic functionality, doing everything you want it to plus some (then ++ some more).

Check out the ack site, or if you are feeling bold, you can install like so:

Make sure you try out the best command-line parameter ever:

E-mail me or shout on Twitter and let me know what you think.

Posted by & filed under Book Reviews.

I work with *nix every day, all day, for the last year.  I’ve always managed to fit those skills into the various jobs I’ve worked, accumulating the experience and skills I’ve needed to accomplish my goals:  Leave the Windows/Mac world behind, and do nothing but work on *nix servers all day.

My *nix -fu has leveled up considerably, but I’m not ready to claim black-belt status quite yet.  I’m always hungry for knowledge. I’m so hungry, that I read books about things that I think I already understand.  Whenever I do so, I always find something new sitting right in the midst of my fundamentals.

When I saw someone in my twitterstream mention ,  it immediately piqued my interest.  Even at the price of $27, I figured I’d give it a try.  I work with Unix processes all day, but I figured there had to be things I didn’t know.  The book was written for Ruby developers and I’m a Perl operations guy, so my major concern was that it would be written in a perspective completely different from my own and alienating.  While there were several moments where it felt the author called all the Ruby devs in a back room and stage-whispered about that funny-looking guy over there, it was nothing hostile and definitely the author was upfront about his biases and perspective.  More importantly, I was still able to learn about.

The introduction had me scared out of my gourd that this was going to be too simple, but the book gets over it quick and starts teaching in short order.  The book seems lean for $27 at 97 pages, but the book has no fat in other than that intro (which isn’t fat –I just didn’t like it :)).  I am a huge fan of tech books just getting to the point.  Technical books need to be as long as they need to be, and not a page more or less.   Any book that avoids page bloat or hyper-compression earns points from me.

The particular use this book had for me was to show me how a developer wants to use the resources on a system I maintain.  That developer perspective I thought would be excluding actually was tremendously insightful.  By talking about the optimizing applications as a developer, I could start thinking about how my developers have built our web application (it’s over five years old so it’s very mature).  I was able to talk to one of our senior developers with a different understanding  and better appreciation today after having read the book.

I think the book is definitely worth a read if you work with *nix-based web systems at all.  The author takes a simple, straightforward approach that distills  good information quickly.  If you pick it up, send me an e-mail or hit me up on Twitter and tell me what you think!

Posted by & filed under Uncategorized.

The smartest thing I ever did was start seeing a psychologist to work on blocks to my productivity.  By many people’s standards, I think I am a busy guy;  I have a lot of stuff going on at any given moment.  I get a lot of stuff done, but for me that’s not enough. I focus, maybe too much, on what it is that doesn’t get done.  I’ve gotten better at filling those gaps overthe years (part of the answer to that is to make sure you aren’t taking on too much –I’ve learned to say no, blissfully, finally) but there is still a part of my life that is searching for more.


More concentration. More purpose. More success.

That sounds rather vague and unspecific, but I’ll get into detailed goals later.  I just wanted to provide you a context for why I am seeing a psychologist besides doing pretty well for myself.  During our latest session, my psychologist revealed something to me that I never knew –it’s common for technologists to hit walls and for that wall to make them feel stupid, or just frustrate them.  Many times (definitely my case) this drives them a little bit to distraction, which can cause late projects, failed projects, etc.

You might be thinking , “Duh!” but in over ten years of tech work I’ve never had this conversation with a co-worker.  Other environments may differ but I’ve never felt that this topic would be safe to discuss at any place I’ve worked. No, better to not speak of it and work extra hours or pull some crazy cowboy stunt to cover up the times you were distracted.  Better to develop some other reason you couldn’t do X or Y.

We’ve worked on some strategies to deal with “the Wall”, but I have to say: thank god it’s not just me!  Just knowing how common it is (my psychologist says she hears it all the time from tech professionals) makes me feel much better.  Not “misery loves company”, but “this is not some rare, uncurable affliction you have”.

Now all I have to do is learn to deal with walls more efficiently.  It’s not that I don’t ever deal with difficulties, or that I give up the first sign of trouble — I’d never have gotten anywhere where that the case — but when approaching topics that are “out of scope”, it becomes for me hard to approach.  Not knowing where even to start begins that wierd tailspin of feeling lostt, then stupid, then….”hey, let’s find something else I feel good and I know that I rock”.  And over time maybe you chip at that wall in bits and spurts.  Over time it gets enough chips that you can see through to the other side, and then you can start to work with it.

But that approach takes forever.  I’ll share some of those strategies as I work with them.