Monday, May 30, 2016

Ruby/Cucumber and vim in multi-project environments...

I'm currently working as a Senior QA Automation Engineer for a major bricks-and-mortar retailer that also has a large online presence. They offer a mobile app, a mobile-website experience, and special support for tablet, Android, and iOS devices. Needless to say, there are a number of different top-level projects, each providing a similar but customized experience for the channel their customers use.

QA automation (QAA) uses ruby/cucumber for automated testing, and the QAA environment provides the 'standard' cucumber structure with features/, features/step_defintions/ for feature files and step definitions. Each top-level project has their own sub-directories so, for example, the mobile-website project uses:
features/mobweb/
features/step_definitions/mobweb
while the tablet project uses:
features/tablet/
features/step_definitions/tablet/
I use vim/tmux for my IDE and depend on vim-cucumber (thanks to Tim Pope for a great plugin) to find relevant step definitions from the feature files. The problem I ran into is that -- because of the multiple project sub-folders -- I was constantly getting the error: "Multiple matching steps found" and thus had to examine each alternative step individually to find the one that related to my project.

This was seriously irritating (not to mention time-consuming) so this weekend I dug into the vim-cucumber code to see what could be done. It turned out to be pretty straight-forward (see my fork of vim-cumber here; I've submitted a pull request and hope it gets merged).

Tim's approach is to first find the top-level cucumber feature file directory (this is in the ftplugin/cucumber.vim file); that could be either features/ or stories/.
let b:cucumber_root = expand('%:p:h:s?.*[\/]\%(features\|stories\)\zs[\/].*??')
That is then saved as a globbable name (e.g., features/**/*.rb):
let b:cucumber_steps_glob = b:cucumber_root.'/**/*.rb'
and later used to find all the relevant ruby files through the use of vim's glob statement:
for file in split(glob(b:cucumber_steps_glob),"\n")
and then iterate through the list looking for the step you are searching for. Nice stuff.

Of course, with the multi-project directory structure we have, there are many 'duplicate' step definitions: we're all tapping on buttons or checking the status of similar things. Think
When(/^I tap on the "(.*)" button$/) do... 
Then(/^I should see the "(.*)" button is (enabled|disabled)$/) do...
as simple examples).

I now have a working solution with a small footprint on the basic plugin. I looked at different ways of doing this, but then hit upon this solution. Export a glob spec to the shell environment variable CUKEFILES. When setting the glob spec, vim-cucumber looks for the existence of this variable and uses it instead of the default value.
if !exists("b:cucumber_steps_glob")
  if empty($CUKEFILES)
    echom "Using default definition for b:cucumber_steps_glob"
    let b:cucumber_steps_glob = b:cucumber_root.'/**/*.rb'
  else
    echom 'Using CUKEFILES environment variable for b:cucumber_steps_glob'
    let b:cucumber_steps_glob = $CUKEFILES
  endif
endif
Once that's set, everything works just as it did before, except with a more precise set of files. Note the slight difference when using CUKEFILES: it assumes that the environment variable already has the /**/*.rb set, so that you could define multiple directories in your glob spec.

The CUKEFILES setting I'm using is:
export CUKEFILES=./features/step_definitions/mobweb/**/*.rb
Now my multiple matches truly indicate duplicate step definitions that I need to examine and probably correct.

Finding a working glob statement was it's own challenge. I wanted to have the option of including more than one directory in the glob spec by using the "|" separator but that's another story

Sunday, May 22, 2016

ES6 highlighting and lint checking in vim

I am a long-time "vimmie" (that is, I use vim for my IDE) and I love the plugin ecosystem which lets me do just about anything I want.

For example, I recently started using ReactJS (wish I hadn't wasted to so much time trying to get Angular to work) and one thing I love about it is how straight-forward it is to setup and start writing TDD/BDD for development (but that's another story).  My configuration uses Babel and es6, so I wanted to get syntax checking and indentation/highlighting working right off the bat.

Here's how -- it's pretty easy. I'm using Vundle as my plugin manager so that's the examples given. If you're using pathogen or vim-plug as your plugin manager, it should be pretty easy to modify and get it running. I've used pathogen and it's excellent, but have no experience with vim-plug.

  1. If this is the first time you've used a plugin, go to the Vundle website and follow the instructions there.
  2. Install the vim-javascript plugin by pangloss: vim-pangloss. This will give you basic syntax-highlighting / indentation using vim's built-in syntax support.
  3. Install the syntastic plugin by scrooloose: syntastic (he's the author of the excellent NERDTree plugin which gives you a hierarchical tree menu for system files). This will allow you to use external syntax checkers with vim, in addition to the built-in syntax checking already available. I use the eslint program for this. The options I use in vim for syntastic are:
    """"""""""""""""""""" Syntastic """"""""""""""""""""""""
    " From: http://usevim.com/2016/03/07/linting/
    set statusline+=%#warningmsg#
    set statusline+=%{SyntasticStatuslineFlag()}
    set statusline+=%*
    let g:syntastic_always_populate_loc_list = 1
    let g:syntastic_loc_list_height = 5
    let g:syntastic_auto_loc_list = 0
    let g:syntastic_check_on_open = 1
    let g:syntastic_check_on_wq = 1
    let g:syntastic_javascript_checkers = ['eslint']
    let g:syntastic_error_symbol = '❌'
    let g:syntastic_style_error_symbol = '⁉️'
    let g:syntastic_warning_symbol = '⚠️'
    let g:syntastic_style_warning_symbol = '💩'
    highlight link SyntasticErrorSign SignColumn
    highlight link SyntasticWarningSign SignColumn
    highlight link SyntasticStyleErrorSign SignColumn
    highlight link SyntasticStyleWarningSign SignColumn
    """""""""""""""""""""""""""""""""""""""""""""""""""""""
    
  4. Install the eslint program. This is a node module installed with the 'global' option so that it's available everywhere as an executable. This is just
    npm install -g eslint
    
  5. You would think this would work (I sure did) but it's not yet fully baked. With this setup, the first import statement in some of your .js files will be flagged as an error. To solve this issue, you have to give eslint a few parameters, in the file ~/.eslintrc.json
    // From: http://eslint.org/docs/user-guide/configuring
    {
        "parserOptions": {
            "ecmaVersion": 6,
            "sourceType": "module",
            "ecmaFeatures": {
                "jsx": true
            }
        },
        "rules": {
            "semi": 2
        }
    }
    
Now you've got full es6 syntax checking and error notifications.

Saturday, March 19, 2016

And I am so done with this, too...

Before I get started, let me just say that I like a lot of things about AngularJS.

The basic approach of using directives in the HTML is very appealing and seems natural; filters are easy and intuitive; custom directives provide great options to adding function with little fanfare; data models fit right in to JavaScript; and the ease with which data flows between the view and the data model is pretty much seamless. Google being behind it in a big way is also a major plus.

But what really excited me about AngularJS was that it had "Testability Built-in" [that's a quote from the AngularJS home page]. I'm a believer in TDD/BDD so that was a big selling point for me.

However, my experience trying to get TDD actually working led me to a different conclusion. Here are some of my experiences.

1. Creating a logger requires what? Logging is simple... right? In Ruby it's just:
logger = Logger.new("my_log.log")  # or STDOUT if you want to make it really simple
logger.error "This is my error"
 
In the Javascript world, it's even simpler:
console.log("This is my error")
 
Sorry, Angular... you make it really ridiculous:
describe('basic test', function(){
    var log;
    beforeEach(inject(function(_$log_){
        log = _$log_;
    }));

    it('should just work', function(){
        log.info('it worked!');
        expect(log.info.logs).toContain(['it worked!']);
    });
});
And notice the crazy little "trick" where you say _$log_ to get this to work. Yeah, it's "documented" but buried away and not exactly readily available.

2. "injector already created. can not register a module"
Hmmm... Angular knows that we want to register a module, they know that the injector has already been created, so why can't they Just Fix It? They do all kinds of other magic behind the scenes; this sounds like an easy one. If they can't Just Fix It, then how about a meaningful error message and documentation explaining why that really happens. This is one of those cases where you have to know the innards of Angular to be able to figure things out.
See SO: http://stackoverflow.com/questions/24900067/injector-already-created-can-not-register-a-module

3. How many different ways are there to 'inject' stuff into your test.. (I sure can't count 'em). In fact, why do you even have to play that game at all? If you look around, you'll find countless ways to inject things...

from: http://andyshora.com/unit-testing-best-practices-angularjs.html
describe("Unit Testing Examples", function() {

  beforeEach(angular.mock.module('App'));

  it('should have a LoginCtrl controller', function() {
    expect(App.LoginCtrl).toBeDefined();
  });

  it('should have a working LoginService service', inject(['LoginService',
    function(LoginService) {
      expect(LoginService.isValidEmail).not.to.equal(null);

      // test cases - testing for success
      var validEmails = [
        'test@test.com',
        'test@test.co.uk',
        'test734ltylytkliytkryety9ef@jb-fe.com'
      ];

      // test cases - testing for failure
      var invalidEmails = [
        'test@testcom',
        'test@ test.co.uk',
        'ghgf@fe.com.co.',
        'tes@t@test.com',
        ''
      ];

      // you can loop through arrays of test cases like this
      for (var i in validEmails) {
        var valid = LoginService.isValidEmail(validEmails[i]);
        expect(valid).toBeTruthy();
      }
      for (var i in invalidEmails) {
        var valid = LoginService.isValidEmail(invalidEmails[i]);
        expect(valid).toBeFalsy();
      }

    }])
  );
});

from: http://stackoverflow.com/questions/12758157/how-to-inject-dynamically-dependence-in-a-controller
var algoController = function($scope, $injector) {
    $scope.base64 = $injector.get('base64');
};
But then there's the 'dynamic' alternative
var algoController = function($scope, base64) {
    $scope.base64 = base64;
};

Or how about this, from: https://www.airpair.com/angularjs/posts/testing-angular-with-karma
function MyController ($scope) {
  $scope.property = 'value';
}
MyController.$inject = ['$scope'];

But then there's this (which won't work with minified code)
function MyController ($scope) {
  $scope.property = 'value';
}
angular.module('myApp', [])
  .controller('MyController', [
    '$scope',
  myController
]);

I'm sure that all these methods work just fine for the folks who documented them (nothing against them; I certainly respect what they can do). But trying to figure out a sane way to do things when you're getting started is... interesting.

On http://stackoverflow.com/questions/31977313/tdd-with-angularjs-and-protractor.
A question I asked with no response...

I could go on and on with the attempts I made to get a reliable, stable approach to injecting code for testing. In the end, I gave up. I'm sure that Angular supporters will scoff and say: "Hey! It's really easy... just do this".

4. So I asked friends who were working with Angular for some help, I got comments commiserating with me about how it's all not very clear, confusing, overly-complicated, difficult, etc. Some folks even pointed to the fact that you have to know a lot about how Angular works internally before you can easily create real tests.

However, the kicker for me was when I asked a member of the Angular core team about testing and his response was a shrug, a pained expression and the comment: "Yeah; we need to work on that."

That's when I said: "Yeah; I need to move on to something else..."

Thursday, October 8, 2015

(Sadly) I am so done with this...

Years ago when I first started working with Ruby on Rails (back in the version 1.0 days) I started off on a Windoze machine... and we all know what a major headache that was. It didn't take me too long to start looking for other alternatives and I settled eventually on Kubuntu.

It took a lot of work to get things up and running but I slowly learned more and more about bash, and setting up Apache, and getting Samba to work on my mostly Windoze network. A lot of work but I eventually got things working and was able to happily develop my system on that platform. I really came to love the power and flexibility that was possible on Linux.

Eventually, I started working with some older MacBooks and while they were relatively under-powered and had their quirks it was a pretty nice setup. And for the past two years now I've been working with current MacBooks and loving it. However, when starting up a new project, I decided to got back and see how Linux would work: it would be my desktop / base machine and I could still use the MacBook when I was on the road.

As usual, there were a number of glitches, not the least of which was that my last version was 12.10 and was no longer supported. Not only does Ubuntu not support the old releases, they actually take down any servers that contain the files necessary for an upgrade so you're pretty much out of luck. I didn't want to start from scratch as I had a lot of things configured. I already had my /home on a separate hard drive so I was eventually able to migrate things over and get up and running on 14.04.

I figured I was good to go... wrong.

Running Rails 4.0 turned out to be a problem: capybara-webkit refused to install so I started researching that. More hassles, more delays, but I figured it would settle down soon.

Wrong again.

I decided to do a reboot and that's when things got really, really bad. I got dropped into a grub rescue prompt, with complaints about a parameter that some well-meaning application had added. And even setting the boot drive (after even more research) didn't solve the problem.

Fortunately I don't have much on the system that isn't backed up -- there's only one small system and database that I'll need to recover but this time I'm going to do it with a live disk and then...

...I am done with Linux. I've come to the opinion that Linux is really great for two kinds of people:
  1. People that want to learn and play and don't mind re-installing frequently.
  2. People that really really know how to make this stuff work well -- sysadmins.
Unfortunately I'm not one of those two kinds: I want to use the system to get meaningful work done, I don't have time to learn everything, and I certainly don't qualify as a sysadmin, even though I've gotten lots of things working over the years.

So... sayonara, Linux... it's been good to know you. You've been a good companion -- albeit often a frustrating one -- and I've learned a lot from you, but it's time for me to move on.

Friday, April 10, 2015

git shortcuts

Over the past few months, I've developed a few shortcuts that make using git a little easier & quicker. Here are a few... I'm curious as to what others are doing. These are defined in my .bashrc

List or use the currently active branch
function gbc() {
if [ -e ./.git ]; then
  git branch | sed -n -e "/^\*/ p " | cut -c 3-99
fi
}
export -f gbc
Now if I want to see what the latest branch is, I simply type `gbc` and the current branch is displayed. What's even more useful, IMO, is that I can now use that function to display the current branch in my bash prompt using something like this:
PS1='${debian_chroot:+($debian_chroot)}\u@\h:[\W-$(gbc)]\w\$ '
and now my prompt looks like this:
jseidel@EDP28:[s_analyzer-master]~/dev/coding/s_analyzer$
I've got mine color-coded (MUCH longer PS1 string) so that the branch name really stands out and I can see immediately where I am.

Quickly add modified (but not new) files
I frequently have a few untracked files in my repo which I do not want to add to the repo, so git adding everything doesn't work for me. So I created a `gadauto` command to do that for me.
function gadauto() {
  git status | ack 'modified:|deleted:' gst.txt | sed -Ee 's/modified:|deleted://' | xargs git add
}
export -f gadauto
Now a quick `gadauto` command adds modified or deleted files to the staging area

Quickly add to and amend a commit
We use git-review and the preferred workflow is to first create a new commit and then amend that commit until you're ready to push it for review. The command to do that is "git commit --amend". To shorten this, I created a simple alias:
alias gcamd='git commit --amend'
Now when I'm ready to add my updated files and amend the commit, all I have to do is type:
gadauto && gcamd
I could shorten that even more but I like to type just a bit more when I could make a messy mistake.

Quickly add merge conflicts that have been resolved
The gadauto command works great but doesn't handle the use case for files that have been conflicted and then resolved during a git rebase, for example. Handling this is just a simple modifaction of the gadauto commend described above.
function gadboth() {
  git status | ack 'both modified:|both added:' gst.txt | sed -Ee 's/both modified:|both added://' | xargs git add
}
export -f gadboth
Now Now following a `git rebase master` and conflict resolution, I just type:
gadboth
and all of the files that were just resolved are added to the staging area, ready for the `git rebase --continue`.

Hopefully this will help someone; I'd be interested to know what your favorite git shortcuts are.

Update If you want to see some really cool git shortcuts, visit Thoughtbot's dotfiles.

Monday, March 9, 2015

No, it wasn't the "world's least helpful error message"

Last wee, I commented about a problem with AngularJS and an error message that I found incredibly unhelpful.  Turns out it was more helpful than I knew although there are still problems.

Here's what it looked like in Chrome DevTools:
But that's a live link, so all you have to do is click on it and you're redirected to an AngularJS page that (hopefully) provides more information. Here's the link, and here's the link text iteself: quite a mouth/link full:

What you get when you click on this link is a customized error page -- pretty neat way to handle things and sometimes it works very well -- this example was spot on:

But here's one that wasn't nearly so helpful:

So...much better than I thought, although there are still some head-scratchers.




Tuesday, December 23, 2014

Trust your auto indentation... really

I knew I liked my editor's autoindent feature, but I didn't realize how valuable it was until this situation popped up. I was reading the book Rebuilding Rails (worth a look) and happened to code a simple method as part of one example.
def call(env)
  (klass, act) = get_controller_and_action(env)
       controller = klass.new(env)
       text = controller.send(act)
  [200, {"Content-Type" => "text/html"},
...
As you can see the indentation isn't correct for the 'controller =' and 'text =' lines and try as I might
vim gg=G
it wouldn't indent properly.  I chalked it up as low priority weirdness to be dealt with later until I loaded up the program and got this error:

undefined local variable or method `controller' for ...

I finally dropped into HexDump mode using vim's nifty little xxd utility
:%!xxd
and got this output for the lines that wouldn't indent properly.
0000060: 6c28 656e 7629 0a20 2020 2020 2028 6b6c  l(env).      (kl
0000070: 6173 732c 2061 6374 2920 3d20 6765 745f  ass, act) = get_
0000080: 636f 6e74 726f 6c6c 6572 5f61 6e64 5f61  controller_and_a
0000090: 6374 696f 6e28 656e 7629 0a20 2020 2020  ction(env).    
00000a0: 20c2 a0c2 a0c2 a0c2 a0c2 a063 6f6e 7472   ..........contr
00000b0: 6f6c 6c65 7220 3d20 6b6c 6173 732e 6e65  oller = klass.ne
00000c0: 7728 656e 7629 0a20 2020 2020 20c2 a0c2  w(env).      ...
00000d0: a0c2 a0c2 a0c2 a074 6578 7420 3d20 636f  .......text = co
00000e0: 6e74 726f 6c6c 6572 2e73 656e 6428 6163  ntroller.send(ac
Notice the 'c2a0' sequence repeated five times in front of the 'controller =' and 'text =' lines. Those non-printing characters were the culprit: the actual variable name was 0x'c2a0c2a0c2a0c2a0c2a0'controller, for example so when I tried to use 'controller', that was still undefined. How those byte pairs got inserted remains a mystery. But I am guessing from this post that it was a non-breaking Unicode character.

Saturday, May 24, 2014

Bash regular expression idosyncracies

As a scripting language, bash can certainly be considered a bit unusual, but it's got the power and the later versions keep adding functionality which makes it easier to get things done. However, even with the latest additions, it's still got its quirks.

Consider the regular expression matching feature (added in version 3)

Given a value
MY_STRING='/usr/local/heroku/bin:/Users/jseidel/bin' 
let's say you want to check for the occurrence of the string 'jseidel/bin'. You might try the following script just to check things out in advance.

#!/bin/bash

export MY_STRING=/usr/local/heroku/bin:/Users/jseidel/bin

if [[ "$MY_STRING" =~ /.*jseidel\/bin.*/ ]]; then
  echo "Found .* version"
fi
if [[ "$MY_STRING" =~ /.*jseidel\/bin.+/ ]]; then
  echo "Found .+ version"
fi
if [[ "$MY_STRING" =~ 'jseidel\/bin' ]]; then
  echo "Found single-quoted version"
fi
if [[ "$MY_STRING" =~ "jseidel\/bin" ]]; then
  echo "Found double-quoted version"
fi
if [[ "$MY_STRING" =~ jseidel\/bin ]]; then
  echo "Found un-quoted version"
fi
What you would find (even in bash 4.3.11) is that only the last one works. 
  1. The first one fails because there are no characters after the 'bin' at the end of the string. This is actually pilot error in that there are no characters following that last 'n' so 'one or more characters' is false.
  2. The second one fails because bash apparently doesn't properly support the '+' operator.
  3. The third and fourth ones fail because bash apparently doesn't properly handle REs in a string. Various docs say the quotes are optional, but that doesn't seem to be the case, either for single- or double-quoted strings.

Took me quite a bit to figure this one out; hope it helps someone else.

Saturday, March 1, 2014

I love Linux, but not Nvidia!

I am really beginning to hate Nvidia! I just spent another 4 hours recovering from an update which confused the Nvidia drivers to no end.  After lots of Googling, rebooting, and testing, here's what worked for me; hopefully it will help others (there certainly are lots of posts out there with similar problems).

For the record, I'm running Kubuntu 13.10, 64-bit, with a GeForce GT630 Nvidia card and have two monitors configured side-by-side through nvidia-settings. Also for the record, the later versions of nvidia-settings actually work very well with multiple monitors, unlike earlier versions which required a great deal of hackery.

Initial symptoms
 After updating some applications I decided to reboot (I am rather paranoid these days about updates and rebooting as I have had numerous Nividia issues) I am able to load KDM (actually lightdm for my version) and after attempting to start my account, KDM drops back to the basic GUI login prompt... which repeatedly keeps being interrupted and returning to login.

The first problem I dealt with was finding and fixing any files in my home directory that were owned by root. This is a well-known problem on Ubuntu (maybe other distributions?) wherein root owns a few files and interrupts KDM loading because of permission issues. To resolve these issues, I ran
find -user root
in my home directory and it listed all files owned by root. Then I ran:
sudo chown jseidel:jseidel *
on the root-owned files to make them mine. If you have links, you may have to use the -h option on chown.

One particular problem file is .Xauthority, which sometimes, some way, gets owned by root and will certainly stop your desktop from loading, either in KDM or Gnome.

What finally worked for me
As I progressed through numerous tests, I noticed that when I dropped into a shell (Ctrl-Alt-F1), I got the message
initctl: Event failed
which can indicate an Nvidia driver issue (I've had these many times before -- always a PITA!)

When I looked in the Kernel log after a failure:
less /var/log/kern.log
I found a series of messages like:
API mismatch...
Nvidia client is version 331.49
Kernel is version 319.60
So... the Nvidia installer goofed! What finally worked for me in this situation was the following:
sudo apt-get remove --purge nvidia*
followed by reinstalling the appropriate Nvidia driver, V331.20 in my case. I do my installations from the command line like so, after downloading the desired Nvidia driver:
cd ~/Drivers
chmod +x NVIDIA-Linux-x86_64-331.20.run
sudo ./NVIDIA-Linux-x86_64-331.20.run

Some useful commands:
less /proc/driver/nvidia/version     # Show installed version

Some possibly useful links:
http://askubuntu.com/questions/41681/blank-screen-after-installing-nvidia-restricted-driver

http://news.softpedia.com/news/How-to-Install-the-Latest-NVIDIA-331-20-Drivers-in-Ubuntu-13-10-399182.shtml

http://ubuntugenius.wordpress.com/2013/11/29/ubuntu-13-10-permissions-fix-this-operation-cannot-continue-since-proper-authorization-was-not-provided-halts-software-updater-shutdown-drive-mounting-dvd-playback-etc/

http://ubuntuforums.org/showthread.php?t=1745841


Monday, January 13, 2014

Getting Phusion Passenger to update the site...

First time using Phusion Passenger on a Rails site that I'm supporting for a new client, hosted on dreamhost.com.

I made a very simple upgrade to a view but it wouldn't display the updated content. If you've worked with Passenger before, you probably know this, but Passenger won't reload any new content unless you update the modification date of a special file "tmp/restart.txt", as in:
touch tmp/restart.txt
This information is buried fairly deep in the documentation so I didn't notice it immediately. Hopefully this will save someone else a bit of head-scratching.



Thursday, October 10, 2013

Ruby variable assignment... how does that really work?

I've recently been working to improve my Ruby skills, so I've started coding many of the exercises in "Cracking the Coding Interview" by Gayle Laakmann McDowell, implementing the problems in TDD/Ruby instead of in Java.

In the process, I ran into a real puzzler -- a bug that I and several friends looked at and couldn't quickly solve. It turned out to be really basic Ruby.

I had developed the following code as part of the answer to CtCI's problem 9.4: Create all the subsets from a set (you can find the complete solution here).
def ss_raw set
  @result_raw << set
  return if set.size == 1
  set.each_index do |i|
    tmp = set
    tmp.delete_at i
    ss_raw(tmp)
  end
end
The intent of this code was to recursively remove one element from the input and generate the subsets. The tmp variable was intended as a proxy for the set variable so that the each_index block could process each subset by deleting one element at a time. It didn't work and gave some really perplexing results, apparently replacing the contents of @result_raw rather than appending to it.

Unfortunately, I forgot the basics:
tmp = set
doesn't create a new variable, it simply creates a new pointer to the same location in storage that set points to. Therefore, modifying tmp also modifies set and the whole process goes south. The solution? Just say
tmp = set.clone
and you get a new and separate variable and the method works as expected.

In many cases, an oversight like this won't cause a problem, but in this cae it wound up modifying the method parameter as well as modifying the basis of the each_index block... a definite no-no and exactly what I was trying to avoid in the first place.

Hopefully this little reminder will help someone else avoid the pain.

Friday, September 6, 2013

RVM oddity

Even though I don't use all the features, I love RVM... it generally makes my life easier when dealing with new/multiple versions of Ruby and/or different gemsets. Definitely recommended, although I know there are people who swear by rbenv. In either case, they both make magical things happen by working in the background... all terrific as long as it works. But when it doesn't work as expected, it can really be difficult to diagnose and resolve.

Case in point. The project I'm working on is done in Ruby 1.9.3-p327 and I have a 'Rails3213' gemset that I've been using. I've always used a .rvmrc file (analogous to .vimrc or .bashrc) to specify my desired ruby and gemset versions on a per-project basis. Somewhere along the line, I upgraded RVM and then get a prompt that I should switch from .rvmrc to .ruby-version for my ruby/gemset information, which I followed.

Mistake. Unbeknownst to me, the information in my .ruby-version file was silently ignored and I was using gems from another location inside RVM.  It only came to light when I encountered some strange test results and then debugging changes I made to a gem seemed to disappear. I finally got a clue of what was going on through a tell-tale signal from vim's "tag jump" (Ctrl-]) command : I had been working on the gem in:
~/.rvm/gems/ruby-1.9.3-p327@rails3213/clearance-1.0.1
but when Ctrl-] I jumped to a method definition inside vim, I was routed to a file in:
~/.rvm/gems/ruby-1.9.3-p327/clearance-1.0.1
Going back to the RVM site, I saw that .rvmrc was still (probably always had been?) the preferred way to define things, so good-bye .ruby-version, hello .rvmc, and things were back to normal.

Lesson Learned. For the future, if I run into such strangeness again, I'll look to my environment much earlier to make sure I'm working with the right set of gems or rubies or whatever.  One thing I was reminded of that I had forgetten was that rvm nicely tells you the ruby & gemset that you're using when you enter a project directory with a .rvmrc file:
Using /home/jseidel/.rvm/gems/ruby-1.9.3-p327 with gemset rails3213
And if I need to check up on things, a quick :
rvm list && rvm gemset list
tells me all I need to know that I'm in the right place:
rvm rubies
   ruby-1.9.2-p290 [ x86_64 ]
=* ruby-1.9.3-p327 [ x86_64 ]
   ruby-1.9.3-p448 [ x86_64 ]
   ruby-2.0.0-p0 [ x86_64 ]
   ruby-2.0.0-p247 [ x86_64 ]

# => - current
# =* - current && default
#  * - default
gemsets for ruby-1.9.3-p327 (found in /home/jseidel/.rvm/gems/ruby-1.9.3-p327)
   (default)
   global
   rails3020
   rails3212
=> rails3213
Another thing I'm going to do more carefully is re-create my tags file whenever there are gem changes so that I'm always up-to-date. I found a great bash one-liner (if I can remember where I found this, I'll come back and give credit):
ctags -R . $(bundle show --path)
This puts all your gems into the tags file which means that you can Ctrl-] on methods that are included in Rails or other gems.

Update:
Now when I enter the project directory, I get the RVM warning:
You are using '.rvmrc', it requires trusting, it is slower and it is not compatible with other ruby managers,
you can switch to '.ruby-version' using 'rvm rvmrc to [.]ruby-version'
or ignore this warnings with 'rvm rvmrc warning ignore /home/jseidel/Dev/MyPasswords_Upgrade/.rvmrc',
'.rvmrc' will continue to be the default project file in RVM 1 and RVM 2,
to ignore the warning for all files run 'rvm rvmrc warning ignore all.rvmrcs'.
It may be slower (I sure can't tell the difference), but I'm going to stick with .rvmrc as long as I can.

Friday, August 30, 2013

Another take on WickedPDF, wkhtmltopdf, and heroku...

A while back, I commented on installing WickedPDF/wkhtmltopdf to work on Heroku. But there are still a few more things to be done to get it all to work. Here's a checklist I provided to a friend when they were having problems.

1. Added gem "wickedpdf" in my Gemfile

2. Copied the appropriate wkhtmltopdf binary into the bin/ directory in my Rails app

3. Updated config/initializers/wicked_pdf.rb to include:
WickedPdf.config = { :exe_path => "#{Rails.root}/bin/wkhtmltopdf" }
so that it knows where I've stored the binary

4. Added the following to the show method in my controller; this is what drives the whole PDF creation. There are many more options available; this is just my minimal setup.
format.pdf do
  render :pdf => "AccRpt_#{@accident_report.id}",
    :template => 'accident_reports/show', :formats => [:pdf], :handlers => [:haml],
    :show_as_html => params[:debug].present?,
   # allow debuging based on url param
    :page_size => :letter,
    #:debug_javascript => true,
    :footer => {
      :left => "#{Time.now}",
      :center => "Accident Report ##{@accident_report.id}",
      :right => "Page [page] of [topage]",
      :line => true         },
    :user_style_sheet => "/assets/stylesheets/pdf.css",
    :layout => "pdf.html" #false end

NOTE The "show_as_html" option. By adding this, you can add "&debug=1" to your URL and see what the html would have been. Sometimes helpful in tweaking the output.

5. Created a new stylesheet called pdf.css and a new javascript file called pdf.js. They don't do a lot, but I pretty much extract out as much stuff as I can so that only what's absolutely necessary is included here.

6. Created a new layout called pdf.html.haml. Cuts out all the overhead associated with 'normal' html display. Again, nothing but what's absolutely necessary to produce the PDF.

7. In that layout, used a Content Delivery Network (CDN) for all the jquery stuff I use - that's directly available to wkhtmltopdf since it's a full URL (Remember I said that you can't use relative URL's with wkhtmltopdf).

8. Where necessary, used the WickedPDF-provided helpers to include other stylesheets/javscript files. Make sure you read the "Usage Conditions - Important!" section on the WickedPDF page - I added that after I ran into problems.

9. Created a show.pdf.haml view template with the data/formatting I wanted.
*NOTE**: You can render an existing html partial as part of your PDF output, but to do so you MUST add:
":handlers => [:haml], :formats => [:pdf]"
to the render and then you can actually reuse an html partial, such as _body.html.haml. If you don't do that, you'll get a complaint that it doesn't have the proper pdf template.

10. Added 'PDF' links at various places, such as in the index view and the show view.

11. Made sure that the PDF mime-type was registered:
'Mime::Type.register "application/pdf", :pdf' in config/initializers/mime_types.rb  

I believe that covers it all.

Wednesday, July 24, 2013

Setting up Jasmine for JavaScript tests

Testing your JavaScript with Jasmine is A Good Thing, so I figured it would be easy to get started. There's great documentation at the Github/Jasmine site as to how the tests are described and what matchers are available, but there didn't seem to be any clear-cut instructions on how to get the whole thing to run once you've installed it. There's talk of a file 'SpecRunner.html' and there were a number of posts asking "Where's my SpecRunner file?". And even after you install the gem, there's no documentation provided with rake -T to tell you what to do.

Turns out it's really drop-dead simple so - just in case someone else has this problem - here's what I did.
cd /path/to/your/project
gem install jasmine
rake jasmine
http://localhost:8888 in your browser
The tests in your project are run automatically.

Wednesday, June 26, 2013

Silence is NOT golden...

Here's a sneaky little routing gotcha.  Let's say you're TDD'ing a new feature and your test requires a 'new' action. Following best practice, you start by saying
resources :accounts, only: [:new]
instead of
resources :accounts
Before we get to the gotcha, why would I say 'best practice' is to avoid the complete definition? Well... remember one of the core principles of TDD:
Write ONLY enough code to get the test to pass.
This means that adding only a single action is all I need do to get the test to pass, so that's all I add (even though I actually write more code in that case). This idea certainly isn't new to me; for example, check out the Thoughtbot TDD class if you're interested in learning more: great resource!

On a broader scale, assuming you've got this whole feature written, you still want to avoid adding any unused actions. If you use every RESTful action except for :delete, then you don't want to allow the delete action to exist, as it offers one more opportunity for the bad guys to attack your site.

Back to the gotcha. Let's say that you code the above example, but - oops - you make a typo and actually code:
resources :accounts, only: [:newt]
Guess what - this error is silently ignored. No route is generated and you don't get an error message. In fact, the only: and except: options just ignore anything that isn't on the list of RESTful actions (index, create, new, update, edit, show, destroy) and give no warning.

It's easy to overlook this as we're used to having Rails give us helpful error messages if we code something incorrectly. Not so in this case.

Sunday, June 23, 2013

"Can't find generator help."

Here is a tricky little case of  mis-direction. I've gotten used to github, heroku, and a host of other command-line tools that allow you to add the keyword 'help' to the command to get some basic usage info. Not so, Rails.  If you type the following inside a Rails application (I'm on Rails 3.2.13 or 3.2.12):
rails generate help
you get the decidedly unhelpful message:
Could not find generator help.
Rails thinks you're looking to generate a "help" something or other.

However, if you type the same command outside a Rails app, you get what you'd expect:
Usage:
  rails new APP_PATH [options]

Options:
  -r, [--ruby=PATH]              # Path to the Ruby binary of your choice
                                 # Default: /home/jseidel/.rvm/rubies/ruby-1.9.3-p194/bin/ruby
  -b, [--builder=BUILDER]        # Path to a application builder (can be a filesystem path or URL)
  -m, [--template=TEMPLATE]      # Path to an application template (can be a filesystem path or URL)
      [--skip-gemfile]           # Don't create a Gemfile
      [--skip-bundle]            # Don't run bundle install
  -G, [--skip-git]               # Skip Git ignores and keeps
  -O, [--skip-active-record]     # Skip Active Record files
  -S, [--skip-sprockets]         # Skip Sprockets files
  -d, [--database=DATABASE]      # Preconfigure for selected database (options: mysql/oracle/postgresql/sqlite3/frontbase/ibm_db/sqlserver/jdbcmysql/jdbcsqlite3/jdbcpostgresql/jdbc)
                                 # Default: sqlite3
  -j, [--javascript=JAVASCRIPT]  # Preconfigure for selected JavaScript library
                                 # Default: jquery
  -J, [--skip-javascript]        # Skip JavaScript files
      [--dev]                    # Setup the application with Gemfile pointing to your Rails checkout
      [--edge]                   # Setup the application with Gemfile pointing to Rails repository
  -T, [--skip-test-unit]         # Skip Test::Unit files
      [--old-style-hash]         # Force using old style hash (:foo => 'bar') on Ruby >= 1.9

Runtime options:
  -f, [--force]    # Overwrite files that already exist
  -p, [--pretend]  # Run but do not make any changes
  -q, [--quiet]    # Suppress status output
  -s, [--skip]     # Skip files that already exist

Rails options:
  -h, [--help]     # Show this help message and quit
  -v, [--version]  # Show Rails version number and quit

Description:
    The 'rails new' command creates a new Rails application with a default
    directory structure and configuration at the path you specify.

    You can specify extra command-line arguments to be used every time
    'rails new' runs in the .railsrc configuration file in your home directory.

    Note that the arguments specified in the .railsrc file don't affect the
    defaults values shown above in this help message.

Example:
    rails new ~/Code/Ruby/weblog

    This generates a skeletal Rails installation in ~/Code/Ruby/weblog.
    See the README in the newly created application to get going.

Thursday, June 20, 2013

wkhtmltopdf and all those gems...


Looking back on my wkhtmltopdf / Heroku experience, there are several take-aways for me.

First off, I had contacted Heroku technical support for assistance, and got back a quick reply:

Unfortunately we don't support installing additional libraries or binaries to our stacks. The best workaround is to vendor these into your project. You'll need to use 64-bit Linux versions to make them work on Heroku; compiling statically can also help ensure that any dependencies needed are included. Similarly, for gems that depend on external libraries, we recommend compiling the gem statically and vendoring it into your project (which is what it seems you have already tried to do).

We realize this is not a trivial task and can be very difficult to get working, and we hope to provide an easier way to do this in the future. Unfortunately we do not have an ETA on when this improved functionality will be available.

If you do wish to try to vendor, or if you have no success with vendoring, your binary, library, or gem, you can use Heroku as your build environment. One of our engineers created a build server that allows you to upload source code, run the compilation step, and then download the resulting binary. You can find this project on Github.
Kinda makes sense to me: they have to protect their stack. I'm not surprised, just a wee bit disappointed that they don't make something as widespread and generally useful as wkhtmltopdf available as a standard binary. Oh well; they do seem to be looking into a better solution, and they did provide some good information. The link to that project looks like it might be useful in the future.

Once I got my app with PDF working, I submitted a documentation update to WickedPDF, which was promptly merged (thanks @unixmonkey). Hopefully that'll provide some assistance to other folks.

One of the problems I had with the various gems that include binary versions of wkhtmltopdf was that they hadn't been updated recently... sometimes in over a year.  So I figured that I should be a good citizen and update them with the latest version (0.11.0 rc1 at the time I'm writing this).  To that end, I just submitted a pull request for the wkhtmltopdf-heroku gem. It purports to automagically support WidkedPDF, PDFKit, and wisepdf, depending on which one is loaded (I've not tested it myself).

However, when I started looking at the wkhtmltopdf-binary "family" of gems, it was pretty involved: steerio/wkhtmltopdf-binary (the one I originally investigated), is forked from unixmonkey/wkhtmltopdf_binary_gem which in turn is forked from michaelcollas/wkhtmltopdf_binary_gem, which is itself forked from zakird/wkhtmltopdf_binary_gem and who know how many forks there are on side legs...

That's not something I'm inclined to sign up for! And given the simplicity of adding the single binary in my app (now that I know how to do it), I'm inclined to keep it that way and avoid any of those gems entirely. In fact, I'm beginning to come to the conclusion that it may be better in the long run to install things manually for what I'll call "minor" gem functionality [Note to self: come up with better terminology for this situation] such as this one: I know more of what's going on and have more control over the final outcome.

It also reminds me that, while we work hard to DRY up our code, the proliferation of similar gems on the web is anything but DRY.

Friday, June 14, 2013

WickedPDF, wkhtmltopdf, and Heroku...a tricky combination

At least for me as it took a bit of twiddling to get it running, first in development and then on the Heroku stack. But with a lot of testing and some help from unixmonkey and my friends at Thoughtbot, I finally got it working. Here's how.

First off, I'm running Rails 3.2.13, WickedPDF 0.9.6, and wkhtmltopdf 0.11.0 rc1, running development on a Linux (Kubuntu 10.04) 64-bit machine. NB: The machine you're developing on and deploying to can be different but then you'll have to have two (or more) different copies of wkhtmltopdf and properly configure WickedPDF to handle that difference. I cover that later in this post.

Install wkhtmltopdf

The download site for wkhtmltopdf has pretty much everything you need for whatever machine architecture you are working with. Download the matching flavor for your development and production machines. If you have only a single machine architecture, then just call your one binary 'wkhtmltopdf'.

If you're working with multiple architectures, label each one appropriately, either with the architecture suffix (e.g., -amd64) or simply '-dev' and '-prod'.

Place the one or more binaries in the directory "/bin", right off your root directory (e.g., "#{Rails.root}/bin") and mark them as executable (chmod +x on linux/mac). Note that for Windows development, you'll have a wkhtmltopdf.exe file for development and probably plain 'ol wkhtmltopdf for production.

Install WickedPDF

Installation is straight-forward; just follow the instructions here. However, there are some critical changes you must make for generating PDFs successfully. The section titled "Styles" gives you the key:
You must define absolute paths to CSS files, images, and javascripts
I'm highlighting this because I missed it the first time around and it cost me dearly.  This section goes on to suggest that you use the helpers that are provided. They are:
wicked_pdf_stylesheet_link_tag
wicked_pdf_image_tag 
wicked_pdf_javascript_include_tag 
Every external file that you use must be referenced with an absolute address, so the normal Rails helpers do not work in many cases [Technical Note: the wicked_pdf helpers simply embed the actual CSS or Javascript in the page]. To make life easier for me, I did the following:
  • Create a special pdf.html.haml layout for all PDF pages which uses the wicked_pdf helpers
  • Use the jQuery CDN for including that set of libraries; the Rails-provided helpers do work here because I'm providing an absolute URL; nothing relative to the Rails root of my application 
= javascript_include_tag "http://code.jquery.com/jquery-1.10.0.min.js"
= javascript_include_tag "http://code.jquery.com/ui/1.10.3/jquery-ui.min.js"

Now here's the strange part [I believe this has to do with the Rails asset pipeline; I'll update this post once I nail that down]. I also had to add all my CSS and JS files to my application.rb file for inclusion in the asset pipeline, thusly:
config.assets.precompile += ['blueprint/screen.css', 'jquery.timepicker.css', ...]
Only with this incantation did everything work properly.

Configure WickedPDF for Development & Production

If you're developing on Linux 64-bit, then configuration is easy as you'll use the same binary for both development and production. Update your WickedPDF initializer file ("config/initializers/wicked_pdf.rb") as follows:

WickedPdf.config = { :exe_path => "#{Rails.root}/bin/wkhtmltopdf" }
If you're developing on something other than Linux 64-bit, you'll have to test for the environment and set your exe_path appropriately.. something like this:
WickedPdf.config do |config|  
  if Rails.env == 'production' then
    config.exe_path = Rails.root.to_s + "/bin/wkhtmltopdf"
  else  ### Following allows for development on my MacBook or Linux box
    if /darwin/ =~ RUBY_PLATFORM then
      config.exe_path = '/usr/local/bin/wkhtmltopdf' 
    elsif /linux/ =~ RUBY_PLATFORM then
      config.exe_path = '/usr/bin/wkhtmltopdf' 
    else
      raise "UnableToLocateWkhtmltopdf"
    end
  end
end

For windows, you'll have to check out the RUBY_PLATFORM values you see and modify appropriately; my windows machine reports "i386-mingw32'.

 If you need different versions of the same architecture (say for final testing a new version of wkhtmltopdf before rolling into production), then just label it with the version number and update your WickedPDF configuration file.

Monday, June 10, 2013

Chrome / PDF Problem...

I've recently been working to add PDF generation to a client site and got WickedPDF working. I ran into a problem which I finally resolved (RTFM very carefully)... got some great help from the Wicked folks along the way. In any event, I was finally generating my PDFs fine with Firefox, but Chrome kept failing with the message:
Sorry, we were unable to find the document at the original source. 
Verify that the document still exists.
You can also try to download the original document by clicking here.
All that had to be done was to click on the 'here' link provided, but that's less than an optimal user experience... Not Good, Google! There are quite a few posts out there with the same problem. I finally figured out that this was caused by the Google Docs PDF/PowerPoint Viewer (I was running version 3.10). Easy solution: in Chrome, go to:
Customize | Tools | Extensions
and disable that extension. Now PDFs are properly displayed.

Tuesday, May 28, 2013

Fixing a database problem on a Heroku app

Recently, I was working on an upgrade to a Heroku app and found to my surprise that my schema_migrations table was wrong in production (horror, actually... I hate this kind of problem because of the possibility of data corruption). The database tables were correct, but the schema was behind by 11 migrations. Suffice it to say I have no idea how that happened, but it must have occurred when I was upgrading from a staging instance to production earlier this month [Note to self: figure out a reliable methodology for doing this].

There are a number of posts that recommend directly updating the Heroku database; this one, for example. I started to take this approach and then realized that there was a safer way to make this change. Heroku has a set of database commands that allow you to copy your production database down to your location machine or copy it back to production. This is better, IMO, than mucking with production data live... even with a backup. Working on your local machine, you can run your tests and generally use all available tools to make sure the modifications work before you upload it back to production.

The steps are dead simple:
  1. Find the id of your production database:
    heroku config --app [name]
    the id you want is usually something like "HEROKU_POSTGRESQL_PINK_URL"where "PINK" will probably be some other color.
  2. Make a backup of your production database using the database id:
    heroku pgbackups:capture --app [name] [database id]
  3. "Pull" the production database down to your local machine (Heroku drops it into your development database):
    heroku db:pull --app [name]
  4. Make your changes
  5. Run your tests
  6. "Push" the modified data back to production:
    heroku db:push --app [name]
  7. Open your browser and run a smoke test on your updated database.
Mission accomplished.