Inotify + Node + FTP = Easy-mode remote dev work
Pre-ramble
Haven't updated my blog in a while, but I had a fun little win tonight that I thought I'd share with the lovely readers of this blog (you know I love all 3 of you).
I am currently working fulltime at Wotif.com in Brisbane, and I'm extremely fortunate to be immersed in the weird and wondrous world of Groovy/Grails for the majority of application development I'm involved in. To make things even more awesome-er we're currently working a on a project that uses CouchDB/ElasticSearch as our primary data provider. Life is bliss. Well, mostly.
The Problem
I still have work come in occasionally from a long-time client I respect enough to give some time to on the weekend. Unfortunately this work is in the form of PHP4 code most of the time. And no, the pain doesn't just end there, usually the work is on a fairly large production Joomla 1.0 site. Needless to say, the technical debt flows freely.
Anyway, this particular site is a real pain to work on, as it's not really in a state where it can be run up locally without a fair amount of pain and misery. I haven't done much work for this particular client in the last few months, and thus I didn't really have a proper LAMP stack running on my machine at home (nor do I really want to). A more recent project I'd done for this client last year was a Joomla 1.5 site, where I had the luxury of "doing it right" - I had set the project up to easily be run up locally using some bash scripts, UnionFS, and a little bit of voodoo. But no such win was to be had for this Joomla 1.0 site.
The work I needed to perform on the Joomla 1.0 site was considerable enough that the prospect of firing up Filezilla and manually FTPing changes was unbearable. In the past I've used Eclipse with an obscure plugin called ESFTP to push my changes to the server as I develop. However this still has an obnoxious required manual step of clicking a button every time I want to push a file to the server.
I figured there had to be an easier way. Then I remembered seeing some cool inotify stuff in Node.js a while ago.
The Solution
I thought to myself: "Wouldn't it be cool if I had a little Node app running that monitored my project on the filesystem and FTP'd changes to the codebase as I made them?".
So I decided to cook something up. A couple of hours later I came up with this:
https://gist.github.com/1652663
I didn't end up using the libinotify bindings for Node.js, as it was a little too low level for a quick prototype. The main pain point was the fact that inotify isn't actually recursive, so you actually have to put together your own code that recursively creates watch descriptors for the directory structure, glue in new watch descriptors as new directories get created, and delete old descriptors as directories disappear. I instead opted to use the awesome inotifywait tool (which comes from the inotify-tools package in Ubuntu) which handles all the un-fun parts of inotify and instead sends nice little status updates on stdout.
Oh, and I was getting bizzare issues with the node-ftp library from NPM, so I just grabbed the latest from the git repo and threw it in with the script.
So now I just fire up this script with the FTP details and paths. It just sits there patiently and creates/deletes directories as needed, and pushes file changes/deletions as they occur.
Now this could definitely have easily been done using pretty much any language, but I think it's a pretty neat and elegant little CoffeeScript/Node solution
Speaking of Node/CoffeeScript, I've been working on a little project in all the spare time I can get. I'm excited to blog about some of the cool stuff I've found/done in that regard in the coming weeks!
That's all for now, internets.
Listening for end of response with Node/Express.JS
I'm currently working with CoffeeScript, Node, Express, and Redis to deliver on a quick'n'easy contract I've been put in charge of. This is the first time I've used any of these technologies in a proper commercial deliverables type project, and I have to say, it's been an absolute delight.
An issue I ran into was I wanted to reduce the boilerplate on handling requests, so I wrote a quick route middleware in Express to create a client connection to Redis, assigning the connection to the request object for easy use . I also wanted to be clever and have this same middleware clean up after itself when the request ended. That is, I wanted the middleware to QUIT the Redis connection when the response had been sent.
Consulting the Express/Connect/Node docs yielded no clues as to how to do this, the closest hint I found was from the Node docs indicating that a http.ServerResponse is a WritableStream. I noticed WritableStreams had a "close" event that is supposed to be called when the Stream is no longer writable. I assumed that if you call .end() on a response then it should trigger this event, so my initial middleware looked like this:
setupRedisClient = (req, res, next) =>
req.redisClient = require("redis").createClient()
cleanup = =>
console.log "it worked!"
req.redisClient.quit()
res.on "close", cleanup
res.on "error", cleanup
next()
I tried using this middleware in a route, and was sad to see that the event was not being triggered.
As a last resort I started digging through the Node source, and lo and behold! I found what I was looking for in lib/http.js. Turns out when you .end() your http response, it will emit a "finish" event.
Now my Redis middleware looks like so:
setupRedisClient = (req, res, next) =>
req.redisClient = require("redis").createClient()
cleanup = =>
req.redisClient.quit()
res.on "finish", cleanup
res.on "error", cleanup
next()
Incoming routes that need a Redis connection simply add this middleware, and hey presto! They can use req.redisClient to their hearts content. Once a response is sent back to the client, or an error occurs with the request, the Redis connection will be cleaned up automagically! Hurrah!
No.de coupon
Huzzah!
Got my no.de coupon in the mail a couple of days ago.
Now I just gotta figure out what I wanna host at http://sammeh.no.de/....
Creating a proper Buffer in a Node C++ Addon
Despite the wordy title, it's actually a fairly simple problem, with a fairly simple solution.
Let's say you have some binary data you want to provide to Node Javascript. No problem, Node has Buffers for that. Digging through the Node.js source code, you find node_buffer.h, which promises a utopia of an ObjectWrap goodness; you can even memcpy your binary data directly to it using Buffer::Data(bufferObject).
"Fantastic! I'll rock one of those buffers and simply return `bufferObject->handle_`!", I hear you exclaim. Not so fast stud.
If the client were to use this Buffer, they'd get a nasty surprise. It's not a Buffer. You see, Node.js has re-implemented Buffers since 0.2. The Buffer you're playing with from node_buffer.h is actually a SlowBuffer. As the name implies, it's working directly on the heap-allocated memory chunk, so alot of operations on it are quite inefficient. Worse still, the interface provided on SlowBuffer is actually different to the Node.js documentation. Allow me to explain.
The Buffer you're used to dealing with from Node.js user code actually originates from buffer.js. These Buffers are actually just "views" on a proper SlowBuffer, so operations like slicing are literally as quick as allocating a new Buffer object that views the SlowBuffer at a different offset and max length.
So how do you create one of these badboys from C++ to pass directly back to JS calling code? Glad you asked. Like so:
// Some data we want to provide to Node.js userland code.
// This can be binary of course.
const char *data = "Hello world!";
int length = strlen(data);
// This is Buffer that actually makes heap-allocated raw binary available
// to userland code.
node::Buffer *slowBuffer = node::Buffer::New(length);
// Buffer:Data gives us a yummy void* pointer to play with to our hearts
// content.
memcpy(node::Buffer::Data(slowBuffer), data, length);
// Now we need to create the JS version of the Buffer I was telling you about.
// To do that we need to actually pull it from the execution context.
// First step is to get a handle to the global object.
v8::Local<v8::Object> globalObj = v8::Context::GetCurrent()->Global();
// Now we need to grab the Buffer constructor function.
v8::Local<v8::Function> bufferConstructor = v8::Local<v8::Function>::Cast(globalObj->Get(v8::String::New("Buffer")));
// Great. We can use this constructor function to allocate new Buffers.
// Let's do that now. First we need to provide the correct arguments.
// First argument is the JS object Handle for the SlowBuffer.
// Second arg is the length of the SlowBuffer.
// Third arg is the offset in the SlowBuffer we want the .. "Fast"Buffer to start at.
v8::Handle<v8::Value> constructorArgs[3] = { slowBuffer->handle_, v8::Integer::New(length), v8::Integer::New(0) };
// Now we have our constructor, and our constructor args. Let's create the
// damn Buffer already!
v8::Local<v8::Object> actualBuffer = bufferConstructor->NewInstance(3, constructorArgs);
// This Buffer can now be provided to the calling JS code as easy as this:
return scope.Close(actualBuffer);
And that's all folks!
Facebook Platform updates – w00t!
According to this blog post over at developers.facebook.com, some exciting changes are coming to the Facebook Platform.
I'm a little slow on the uptake here, the blog update I'm referring to is now 10 days old. I haven't been in the Facebook dev world of late, so I hope you'll forgive me for regurgitating old news.
The big update for me is in the 4th paragraph:
We are also moving toward IFrames instead of FBML for both canvas applications and Page tabs. As a part of this process, we will be standardizing on a small set of core FBML tags that will work with both applications on Facebook and external Web pages via our JavaScript SDK, effectively eliminating the technical difference between developing an application on and off Facebook.com.
This is excellent! I have actually been holding my breath for something like this for a while now. The restriction of FBML only content for tabs has been extremely restrictive, here are a handful of reasons why:
- Very strict HTML parsing - because the Platform was rendering tabs inline previously, it of course had to be VERY careful in how it handled offsite data, to protect users from all manner of scams/attacks. Now full flexibility is available because your iFrame is yours to control.
- Insanely strict JS parsing - same as first point, application tabs could only leverage basic Javascript and the unwieldy, poorly documented FBJS (Facebook Javascript). Now, you're free to access manipulate your IFrame document/window objects, DOM manipulate, include third party JS libs, etc, all to your hearts content.
- Embedded media was a pain - Granted, the tab FBML *did* allow you to embed Flash and AIR apps etc, but there were countless threads on the forum outlining issues they were having interacting with the host page, etc.
- Tab activation policies - There were some extremely frustrating rules with the way tabs were allowed to be "activated". No JS/FBJS was allowed to execute until the user had interacted with the tab in some manner, such as focusing a form element or clicking somewhere. This made it very cumbersome to implement any meaningful interactions with the user; alot of obnoxious boilerplate code had to be written for various ways in which you may actually start doing anything meaningful from JS, like AJAX requests.
- ... and lots more Everything from CSS parsing to the occasional time where the tab would just sit in an endless loading display when clicked. When I was writing a tab page I remember bashing my head against the wall trying to get some content to sit nicely in a cross browser fashion... it would have been easy under normal circumstances, but the Platform tab flavour was refusing to accept the *display: inline IE hack in the CSS. Joy.
This is going to really open up some great possibilities for interactive, rich web applications. I really cannot wait for this feature to be rolled out.
This news does come with some disappointment however; the sixth paragraph on the developers blog states the following:
Finally, due to low usage rates, we will remove application tabs from user profiles in the next couple months. Application tabs will continue to be supported on Facebook Pages.
There are plenty of great use cases to have application tabs on a user profile page. My personal Facebook profile has a tab that displays my latest last.fm scrobbles, my latest blog posts, etc. Personally, I believe that if the adoption rate for user profile tabs is low, the Facebook team should be coming up with ways to increase user acceptance of this feature, rather than removing it all together. Besides, the functionality in Facebook Page tabs is pretty much identical to the user profile equivalents... Why not support both?
There's other goodies in the developer blog update too, such as cleaning up the REST API considerably.
All in all, exciting changes coming to the Facebook platform in the coming months!
Internet Explorer 9…. wow?
Okay, so you'll never make a IE convert out of me. I think I speak for the web developer community as a whole when I give Internet Explorer and Microsoft a great big middle finger salute.
HOWEVER! After downloading the Internet Explorer 9 Platform Preview, I got the shock of my life. An Internet Explorer that is taking a DECENT effort in supporting web standards that are almost as old as myself?! Could this be?!
Seriously, their new SVG implementation is amazing. HTML5 support. Hardware accelerated Javascript processing and DOM rendering. CSS3? BORDER-RADIUS SUPPORT?! This is certainly not the Internet Explorer I've come to love to hate.
Of course, I can easily dismiss all of this with the simple statement "5 f*****g years too late, chump". However I can't bring myself to do so, when I am continually reminded of the fact that this family of browsers still makes up a stupid amount of the market today. Yes, IE share is slipping, but I truly wonder if we'll ever see that glorious day where IE drops below 40-50% market share. If the migration of standard consumer PCs ensures that we see a good portion of users running on IE9 rather than the joke we've come to call IE7, or the slightly less humerous joke we've come to call IE8 ... well, that's certainly the lesser of two evils isn't it?
Unfortunately, from what I've read, IE9 probably won't be around for another year or so. So in the meantime, I'll continue to burn baby animals on the IE altar, hoping the gods of Microsoft will hear my pleas for my mortal webapplication to function correctly in their worthless POS software (I ain't talking about no point of sale here either).
Hopefully in that time they'll improve their laughable ACID3 score. Then again, maybe not. Best not to make people too happy, might start putting their faith in you or something ...
How to access Facebook API from Chrome Extension
So following on from my previous post, I decided I wanted to try two new things at once: Facebook API and a foray into developing a Google Chrome extension. With the notification API that will (hopefully) twig on in other major browsers, I'm writing an extension that will display a desktop notification when a Facebook notification is received. All that will be required is for Chrome to be running.
Step 1 was to play around with Chrome extensions. I haven't worked with plugins/extensions in any other browser, but I have to say.... Google NAILED this. It's so intuitive and straightforward. So easy to debug, so much power. I love you Google.
Anyway. Step 2 was where things got interesting - connecting to Facebook via the Javascript API and Facebook Connect. This proved to be quite complex. Facebook Connect has this fascination with the whole cross domain communication channel (XD for shortness) nonsense. Basically, this requirements essentially means to use the Javascript API you need to be hosting it on a standard web server to be able to add the xd_receiver.html file in a public-facing area where Facebook can see it.
After digging around the site for a while, I came across their fancy new Open Source Javascript SDK, I couldn't help but notice this didn't make a mention of the XD setup anywhere, so I assumed perhaps I might be able to work with this system better. As it happens, my suspicions were correct. The new JS library has some crazy voodoo magic set up so that an XD file isn't required, there's some kind of "XD proxy" running on the Facebook servers which will send login session data along the line to the JS library, using either document.postMessage or some Flash workaround thingy (not really sure how that works, don't particularly care.)
I immediately tried to use it in a Chrome extension, but alas! There was a few issues. I came up with workarounds for both though.
Issue #1 - the API does the following in the constructor:
_domain: {
api : window.location.protocol + '//api.facebook.com/',
cdn : (window.location.protocol == 'https:'
? 'https://s-static.ak.fbcdn.net/'
: 'http://static.ak.fbcdn.net/'),
www : window.location.protocol + '//www.facebook.com/'
},
Great! That would be fantastic if our chrome extension was running on http, but the protocol when running from options or background.html pages is "chrome-extension://". So when the API went to make with the server side communications, it was trying to remote to chrome-extension://api.facebook.com, which of course is not really gonna work.
My solution to this was to just fudge it by putting the following snippet in before using the FB lib anywhere:
FB._domain = {
api : 'https://api.facebook.com/',
cdn : 'https://s-static.ak.fbcdn.net/',
www : 'https://www.facebook.com/'
};
With that change, the API won't have any more tantrums when trying to phone home. Easy!
Issue #2 - Facebook Connect is still broken.
This one was the biggie. Essentially, even though the new API does some cool hocus pocus with XD, it's still referencing the "origin domain" in the request, this original domain is of course just going to be the extension URL, which is not terribly useful, as Facebook will freak out when it gets a request coming from an invalid URL. I tried fudging the origin to a valid domain (dodgy I know, I was getting desperate). Interestingly, Facebook was fine with this, but of course when the request came back to the browser and the XD proxy tries to postMessage() the session data back, the browser freaks out as it looks like an XSS attack.
There might be other ways around this drama, but I opted for what I feel to be a fairly elegant solution.
Essentially, I opted to "pretend" I'm something of a desktop application trying to authenticate with Facebook application (which is true in a sense, I suppose). This Developer Wiki page gave me some insight into how to authenticate my application with the FB user the old-fashioned way.
The idea is this: popup the login/app-authenticate page manually with a special URL, setting the return URL to a random dummy Facebook page they have running for desktop app clients: http://www.facebook.com/connect/login_success.html. When the user visits the login page, once they have logged in and allowed the app access (or if they are already logged in and have already allowed the app), they are redirected to a page that has the session data in the querystring encoded in JSON. Login achieved.
So I set about doing this in my extension, simple enough to start with:
var win = window.open("http://www.facebook.com/login.php?api_key=
Great! If I was a *real* desktop application and I was running a Webkit/IE/whatever browser instance as some kind of evil overlord, I could just detect when the browser redirects to login_sucess.html, grab the querystring, parse the session data out of it, and be on my merry way! I'm running in a browser though, so how about I just access the child window that I opened with window.open, access the location.search property and parse that? "NO", says the magical little pixies living inside the browser, "That would be against my strict Same Origin Policy (SOP, kinda like ... sop story, teehee)!!!".
Fair enough. There was an easy enough workaround though, I just embedded a content script via the Extension to sniff out the session data when it became available, and send it to the main extension. Like so:
Add the trigger for the content script to the extension manifest.json file:
"content_scripts": [
{
"matches": ["http://www.facebook.com/connect/login_success.html*"],
"js": ["prototype.js", "intercept_session.js"]
}
],
I threw prototype in there just for convenience sake, as you'll see in the next step.
Then intercept_session.js looks like this:
var params = window.location.search.toQueryParams();
if(!params.session) return;
var session = JSON.parse(params.session);
chrome.extension.sendRequest({message: "setSession", session: session}, function() {
window.close();
});
What this code does is parse the querystring (using Prototype), then check if the session data is present. If it is, parse the JSON into the session object and send it off to the extension Background Page via the extension message passing system to be saved.
The background page simply has this:
var session = null;
if(localStorage.session)
{
session = JSON.parse(localStorage.session);
}
chrome.extension.onRequest.addListener(
function(request, sender, sendResponse) {
if(!request.message)
return;
switch(request.message)
{
case "setSession":
{
localStorage.session = JSON.stringify(request.session);
session = request.session;
sendResponse();
break;
}
case "getSession":
{
sendResponse(session);
break;
}
}
});
Again, pretty straightforward stuff. I'm using the groovy new HTML5 local storage to remember the session data even if the browser is closed. The message handler simply listening for a session to be passed to it. When a session is provided, it will save it to local storage and a local variable. The getSession functionality is so other areas of your chrome extension can retrieve the session as needed (for example if you have a popup and want to query FB from there or something). You could obviously use this session anywhere as needed.
And that's that! From here you can make API calls to your hearts content. There's obviously some important things left out here, stale checking of the session when the background page loads up for example. Also, requesting extended permissions is not covered here, but it's pretty much the same as how the login deal works anyway. You would just update the login_success.html intercept script to check if this was a response for extended permissions, and check the querystring to ensure the permissions were supplied.
I've cooked up a quick little demonstration of this stuffs. Click here to install a demo extension that will add a button to the right of your address bar, which will show some clickable icons of 5 of your friends in a popup. You can also see the code for this extension here.
Time for me to go finish this extension!
Shiny Desktop Notifications from Google Chrome
So I was dicking around in Google Calendar the other day, updating settings and trolling for any cool new Labs stuff to enable... Anyway, I made some changes and clicked submit. I was provided with the ol' yellow InfoBar that asked me for permission to allow something to ... blah blah blah. I was a on a Google site and I have pretty much entrusted my soul to Google already - so I didn't bother reading it before clicking Allow.
Imagine my surprise when a sexy little desktop notification popped up in the bottom right of my screen! Intrigued, I did a bit of Googling, there's a couple of news posts about it, but nothing substantial... other than the design doc hosted on dev.chromium.org.
It's currently only available in the developer channel of Google Chrome. If you're running the dev channel, then check out the example below:
For those interested, that code that powers this little example can be found here.
Very cool stuff. There are so many places this would come in handy. Super Facebook notifications, anyone?
