Automating Light Switches with Node.js

I was originally planning on creating a roguelike for my fun Christmas project, but I decided to create an app to control the lights in my house instead. Home Automation, it’s the future! ūüôā


The app can turn lights on and off, dim to a percentage value, and update UI values in realtime.

If you plan to do something similar, you’ll need a Fibaro Dimmer Module in each light switch, a (local!) server running Node.js, a Z-Wave controller (mine is an Aeotec Z-Stick), the Open Z-Wave library and Socket.IO.

If enough people are interested I’d be happy to write a tutorial, but I think it’s a pretty niche topic – I’d hate to be writing for an audience of one. Let me know what you think in the comments below or hit me up on Twitter.

Twitter Sentiment Analysis with Node.js

Hey folks! This tutorial will demonstrate how to use Node.js – together with Twit (a Twitter API client) and Sentiment (a sentiment analysis library) – to conduct sentiment analysis on the Twitter zeitgeist.

Why? Because it’s interesting! Twitter – despite it’s slow descent into a toxic cesspit – is how you find out what’s happening right now; it represents a global stream of consciousness by way of millions of one-hundred-and-forty character long text snippets.

Using the Twitter API, we can query the Twitter hosepipe for material relevant to our interests. We can then aggregate and interrogate the results and, with the aid of natural language processing tools, divine how people generally feel about something. “Something” could be an event, a product, a brand – anything really. Suffice it to say, marketers love this stuff.

I love indie games, and I’m based in Guildford. Being an avid sci-fi fan, No Man’s Sky is definitely on my radar. So I’m going to use sentiment analysis to find out how people feel about procedural generation in computer games. Let’s get started!


Firstly, you’ll need to create an app in Twitter. This will give you the credentials you need to access and query the Twitter API. Log in to Twitter Application Management, and click the button entitled “Create New App”.

Twitter Application Management Page - Create a New Application
Twitter Application Management Page – Create a New Application

Once you’re all set up, you’ll be directed to your app’s credentials page. You’ll be needing these later.

Using Terminal, create a new folder and create a new JS file in it.

mkdir twitter-app && cd twitter-app
touch app.js

Install Twit and Sentiment;

npm install twit sentiment

Now let’s edit the JS file.

subl app.js

Require Twit and Sentiment, and set up your access credentials.

var twit = require('twit'),
    sentiment = require('sentiment');

var t = new twit({
  consumer_key : 'YOUR_CONSUMER_KEY',
  consumer_secret : 'YOUR_CONSUMER_SECRET',
  access_token : 'YOUR_ACCESS_TOKEN',
  access_token_secret : 'YOUR_ACCESS_TOKEN_SECRET'
});

Okay, now we can query the Twitter API! Let’s search for the phrase (a phrase is indicated by double quotes) “procedural generation.” (N.B: The Twitter Search API only extends to around the past seven days.)

t.get('search/tweets', { q : '"procedural generation" since:2015-10-08', count: 100 }, function(err, data, response) {
  console.log(data);
});

If you run the app – 

node app.js

– you’ll get a JSON object back containing the data for 100 tweets. This is the sanitised output for -one- of those tweets.

{ statuses: 
   [ { created_at: 'Thur Oct 15 14:06:44 +0000 2015',
       id: 000000000000000000,
       id_str: '000000000000000000',
       text: 'text',
       truncated: false,
       entities: [Object],
       metadata: [Object],
       source: '<a href="#" rel="nofollow">@Twitter</a>',
       in_reply_to_status_id: null,
       in_reply_to_status_id_str: null,
       in_reply_to_user_id: null,
       in_reply_to_user_id_str: null,
       in_reply_to_screen_name: null,
       user: [Object],
       geo: null,
       coordinates: null,
       place: null,
       contributors: null,
       retweeted_status: [Object],
       is_quote_status: false,
       retweet_count: 10,
       favorite_count: 0,
       favorited: false,
       retweeted: false,
       possibly_sensitive: false,
       lang: 'en' } ],
  search_metadata: 
   { completed_in: 0.045,
     max_id: 000000000000000000,
     max_id_str: '000000000000000000',
     next_results: '?max_id=000000000000000000&q=%22procedural+generation%22&count=100&include_entities=1',
     query: '%22procedural+generation%22',
     refresh_url: '?since_id=000000000000000000&q=%22procedural+generation%22&include_entities=1',
     count: 100,
     since_id: 0,
     since_id_str: '0' } 
}

“Hosepipe” sounds about right. Let’s whittle this down a bit.

t.get('search/tweets', { q : '"procedural generation" since:2015-10-08', count: 100 }, function(err, data, response) {
  for (var i in data.statuses) {
    if (data.statuses[i].lang === 'en') {
      var s = sentiment(data.statuses[i].text);
      console.log(s.score + ' ' + data.statuses[i].text);
    }
  }
});

This will return the text of 100 tweets, prepended with a sentiment score. A negative score indicates a negative sentiment, a positive one indicates a positive sentiment.


Update July 2017

When I originally wrote this tutorial, the Sentiment was still in it’s infancy (the developers hadn’t yet written the code to handle negation in statements) so I figured I’d re-write this section to reflect the changes. I’m grateful (as always) to FOSS contributors that make libraries like Sentiment happen.

Let’s run the query above again, removing the date modifier, limiting the number results to 10, and filtering out retweets and replies (which can often be taken out of context).

t.get('search/tweets', { q : '"procedural generation" -filter:retweets -filter:replies', count: 100 }, function(err, data, response) {
  for (var i in data.statuses) {
    if (data.statuses[i].lang === 'en') {
      var s = sentiment(data.statuses[i].text);
      console.log(s.score + ' ' + data.statuses[i].text);
    }
  }
});

This is what we get back:

4 like take the procedural generation shit and work on it, when it's good make a good game out of it
3 This GDC talk gives a really good overview on procedural generation ūüôā https://t.co/y16em13WcV
-4 Screw the procedural generation, the core problem here is the none-orbiting planets. You don't know my pain. #NoMansSky
0 Almost there with the procedural generation... #GameDev #IndieDev https://t.co/nXekRxYYgx
-4 Sure, it has procedural generation, I'm fighting back tears because of the shallow gameplay. I'm calling Anonymous. #NoMansSky
0 Got my copy of https://t.co/gWrr7vLPPA today. Should keep me busy for a while …
-6 Ugh... finally got the core of this procedural generation algorithm of mine working... holy shit wasn't that exhausting...
0 GameMaker Studio 2 Action RPG Tutorial - Part 1 - Procedural Generation: https://t.co/1RxY4rws3f via @YouTube
0 I added a video to a @YouTube playlist https://t.co/1RxY4rws3f GameMaker Studio 2 Action RPG Tutorial - Part 1 - Procedural Generation
3 The procedural generation in Dwarf Fortress is really darn impressive. https://t.co/Lz8EWQSLhb

Remember, the first number in each line is the sentiment score; so line 2 reveals a positive sentiment about a GDC talk, and line 3 reveals a negative sentiment regarding procedural generation in No Man’s Sky.

Now, if you’ve been paying attention, you’ll have noticed a problem with my original choice of search phrase. “Procedural generation” is a bit of a broad concept, and we can’t yet determine context from a tweet. Do I mean procedural generation pertaining to a specific game? A specific algorithm? A tool in the Unity Asset Store? A GDC talk? I *could* use a ton of filters to pare down the results – but I think, for now, it’s better to limit searches to proper nouns. Events, brands, products – concrete “things”. Let’s try something else.

I’m trying to keep things positive here, so instead of looking at No Man’s Sky, I’m going to examine sentiment regarding my newest favourite indie game, Caves of Qud:

0 Average Gunslinger On The Roll in Caves of Qud Weekly Run #21 Part 1: https://t.co/ZLdFVZmyfB via @YouTube
3 I think today is going to be a nice chill day with some Caves of Qud and some RimWorld ūüėÄ
-4 the real caves of qud lore question is where are these cannibals finding all these fucking missile launchers
0 When I get addicted in to a game I get really in to them. 96 hours in Caves of Qud. But I think I'm burned out on it now.
0 my dude @Craigoryham is making some VERY spooky tracks for Caves of Qud https://t.co/ZNas6x1Htg
0 I can think of a billion other things I'd rather be doing rn than cleaning this basement.... all of them just happen to be Caves of Qud...
1 caves of qud status juicing cannibal gary: don't shoot a missile in a tiny room juicing cannibal steve: why not
1 Caves of qud is. Oh god not my contacts.
4 Had a lot of fun with Caves of Qud tonight. Definitely need to do more with a beguile build.
5 Caves of Qud is fascinating but so long and with so little narrative variation that I think this'll let me enjoy it… https://t.co/edy2OwVuVI

(Caves of Qud is Dune-inspired roguelike awesomeness. Seriously, go buy it.)

Ten tweets is far too small a sample to be statistically significant. Let’s average the sentiment score over the past week.

t.get('search/tweets', { q : '"Caves of Qud" -filter:retweets -filter:replies since:2017-07-04 until:2017-07-11', count: 100 }, function(err, data, response) {
  var aggregate = 0,
      numScores = data.statuses.length;

  for (var i in data.statuses) {
    if (data.statuses[i].lang === 'en') {
      var s = sentiment(data.statuses[i].text);
      aggregate += s.score;

      // remove any neutral scores - they bring down the average
      if (s.score === 0) numScores -= 1;
    }
  }

  console.log(aggregate / numScores);
});

This returns a score of 1.173913043478261 (remember, any score greater than zero reveals a positive sentiment). So we can say that, over the past week, Twitter users have been generally favourable in their discussion of Caves of Qud. Let’s try another – how is Stardew Valley doing this week? Computer says 0.6052631578947368 – so still generally positive, but not as much so as Caves of Qud. And No Man’s Sky? -0.2625 ūüôĀ

It’s not 100% accurate, as there are a lot of edge cases, and the tools are still being developed. But hopefully you can see how it’s useful to determine sentiment concerning “a thing.” Do you have a product that you update on a regular basis? Now you can gauge whether your latest update was a hit or a miss. Sentiment can be applied to any text – not just tweets. You could use it ascertain average sentiment on user reviews or customer surveys. Why not take my example and extend it? I’d love to see someone do some data visualisation (using D3.js?) with this. Just treat it as a jumping off point. Let me know what you make of it in the comments or on Twitter.


Note: The Twitter Search API has also seen some updates since my original post – it now supports filtering by positive and negative sentiment.


So whilst you can’t use it (without some wrangling) to get an average, you could use it to reveal (and hopefully recify) negative user sentiment about your product. Or perhaps you’ve just shipped a unicorn and want to bask in adulation? :] Perhaps Twitter should just apply this filter globally – though I’m not sure the world needs rose-tinted goggles right now.

Attribution: Google Images can’t find the original creator of the header image used for this post. The closest I could find is from @bskipper27. If you created this image, please let me know so I can correctly attribute your work. Thanks!

Migrating to Express.js v3

I’ve been working on a Node.js web application for a couple of months now. ¬†A few days ago I ran npm update¬†on my project without really thinking about it (d’oh), and my Express.js install was upgraded to v3. ¬†There are a lot of changes in v3, and it took the best part of a day for me to work through them all. ¬†I’ve outlined some of the issues I encountered below. Some of them aren’t particularly easy to find fixes for, so hopefully this will help someone out.

Getting started

Your first port of call should be the migration guide:
https://github.com/visionmedia/express/wiki/Migrating-from-2.x-to-3.x. ¬†There is also a wiki article on the new features (https://github.com/visionmedia/express/wiki/New-features-in-3.x), but there’s no need to read through this until you’ve got everything fixed and working again.

‘Layout’ removed

The first thing that will be immediately apparent is that none of your views are rendering properly.  The default way to render views in Express v2.x was to have a master layout template that all other views inherited from. If you were using Jade as your template engine, a layout template used to look something like this:

!!! 5
html
  head
    title Title
    link(rel='stylesheet', '/style.css')
  body!= body

This template would be applied to all other views – all other views inheriting from this template would be rendered as != body.

Both ‘layout’ and partials have been removed in Express v3, so when you first load your site or application after the upgrade, you’ll see that only the body of your view template is being rendered to the page. ¬†The fix for this is dependent upon which template engine you use. ¬†I’m using Jade, so I’ve outlined the fix below, but if you’re using EJS or similar, you’ll need to look it up.

Layouts in Jade

In the latest version of Jade, the way in which views are rendered has been ‘reversed’ as it were – rather than views inheriting from layouts, a view will specify the layout it wants to extend. This gives you greater flexibility with your views, as you can have different layouts for different purposes.

To get your views to render again, you’ll firstly need to go through all your views and specify the layout you want to extend at the top of each template file, using the keyword¬†extends. The path to layout is relative, so if your views are nested in folders, don’t forget¬†../. ¬†One thing I haven’t figured out yet is if you can set a fixed path to layout, so that any deeply nested views can inherit from extends layout rather than extends ../../../layout. Answers on a postcard please :p

The second thing you’ll have to do is designate blocks in your layout – chunks of code to be replaced with the content in the view template:

// layout.jade
!!! 5
html
  head
    title Title
    link(rel='stylesheet', '/style.css')
  body
    block content
    block scripts

 

// view.jade
extends layout

block content
  h1 Content
  p Some content here 
block scripts
  script(src='/js/lib/jquery.min.js')

This should fix your view rendering problems.

req.flash removed

Req.flash was removed in favour of attaching messages to req.session.messages. ¬†I’ve made extensive use of req.flash in my application, so it would have taken me a long time to rewrite all my message code. ¬†Luckily, the migration guide points us towards¬†connect-flash, a middleware that implements req.flash in Express v3 – very handy if you don’t want to do a big refactor.

Dynamic helpers

Dynamic helpers have been removed in favour of app.locals.use(callback).  Now, instead of assigning view helpers to an object, you assign them directly to res.locals:

Express v2.x

app.dynamicHelpers({
  session: function(req, res){
    return req.session;
  }
});

Express v.3

app.locals.use(function(req, res) {
  res.locals.session = req.session;
});

Socket.io not serving client

There are a number of reasons why socket.io might not be serving the client, and just as many proffered solutions on stackoverflow to try.  After a bit of digging, I discovered that the bug in my application was related an issue with Connect:
https://github.com/senchalabs/connect/issues/500#issuecomment-4620773

Long story short, you have to call app.listen() before¬†io.listen() – this wasn’t the case before. It’s annoying how the simplest bugs can sometimes take the longest time to track down ūüôā

Socket.io and sessions

In my application, socket.io has access to session data. ¬†When I was initially developing this functionality, I leaned heavily on the code from¬†Daniel Baulig’s excellent blog post on passing session information to socket.io via the handshake/authorisation method (http://www.danielbaulig.de/socket-ioexpress/). ¬†As Daniel has now noted at the top of his guide, it requires a few tweaks to work with Express v3.

The first thing that you’ll need to change is where you put your application’s session secret. Formerly, you specified this in express.session – you’ll need to move it to reside in express.cookieparser:

app.configure(function() {
  app.use(express.cookieParser('somesuperspecialsecrethere'));
  ...
});

If you make the above change and run your application, you’ll notice that the session cookie is now being transmitted, but we get another 500 error instead. If we take a look at Daniel’s code:

sio.set('authorization', function (data, accept) {
    if (data.headers.cookie) {
        data.cookie = parseCookie(data.headers.cookie);
        data.sessionID = data.cookie['express.sid'];
        // save the session store to the data object
        // (as required by the Session constructor)
        data.sessionStore = sessionStore;
        sessionStore.get(data.sessionID, function (err, session) {
            if (err || !session) {
                accept('Error', false);
            } else {
                // create a session object, passing data as request and our
                // just acquired session data
                data.session = new Session(data, session);
                accept(null, true);
            }
        });
    } else {
       return accept('No cookie transmitted.', false);
    }
});

With a bit of debugging, you’ll find that the line that starts sessionStore.get¬†is failing and throwing an error on the next line. ¬†This is because the way that cookie signing is implemented in Connect has changed, and sessionStore.get¬†is being passed the long session id from the cookie, rather than the short (24 character) session id it’s expecting (for more information see this thread). ¬†In order to get the short session id, you need to split the long session id at the dot:

data.sessionID = data.cookie['express.sid'].split('.')[0];

Data.sessionID should now contain the short SID Рthis will be evaluated against the SID contained in the sessionStore as normal.  No more pesky 500 errors!

If you have any comments or suggestions, please feel free to comment below.  Thanks!

Book Review: MongoDB in Action

Happy New Year everyone! ¬†I’ve been a bit quiet lately, owing to the holidays and work obligations, but I’m going to make a concerted effort this year to be a bit more active on my blog. ¬†I don’t, however, count writing skills amongst my talents – I can do tutorials, and how-to’s, but I don’t think I’ll ever be able to write informative,¬†expository articles like some members of the JavaScript community do. ¬†I’d rather get my own hands dirty in the code than write about it!

So, in lieu of anything particularly insightful or original, I’ve decided to write some book reviews. ¬†I spend a fortune on books, half of which I never get around to reading. ¬†I’m hoping that creating a self imposed obligation to write book reviews will help me clear the backlog, and that the reviews might be useful to anyone thinking about making a purchase. And if they’re not, well, at least my blog gets some SEO fodder.

Enjoy ūüėČ

Book Review: MongoDB in Action

Author:

Kyle Banker

Publisher:

Manning

Publisher Summary:

MongoDB in Action is a comprehensive guide to MongoDB for application developers. The book begins by explaining what makes MongoDB unique and describing its ideal use cases. A series of tutorials designed for MongoDB mastery then leads into detailed examples for leveraging MongoDB in e-commerce, social networking, analytics, and other common applications.

Review:

I was tinkering with a Node.js application over Christmas, and arrived at a point where I needed to implement a database into my app.  MongoDB seems to be the DBMS of choice for Node developers, so I bought MongoDB in Action to learn a bit more about it.

I wanted a book that would get me through the introductory content fast, and would enable me to be productive in as short a time as possible.  Thankfully, the author had similar goals in mind when writing the book, and split the book up into three parts: Getting Started; Application Development in MongoDB; and MongoDB Mastery.

Part One begins by discussing the origins, history and evolution of the project; and then goes on to discuss why you would choose to use MongoDB over, for example, a key/value store, or SQL. ¬†This is followed by an introduction to the JavaScript shell and some of the core CRUD (create, read, update, delete) operations. ¬†The explanation of the core concepts is accompanied by the development of a simple application with Ruby. ¬†I personally don’t know a lot of Ruby, but the examples were easy to follow, and helped me understand how a similar implementation would work in Node.

Part Two delves deeper into MongoDB’s document data model and query language. ¬†The author explains how to design, create and manipulate documents and collections using the JavaScript shell, and then illustrates how you would implement this in practice by building an e-commerce application. ¬†The example application is quite extensive – it covers¬†products, product categories, users, orders and product reviews. ¬†Any potential issues are highlighted, and any workarounds and solutions are discussed.

Part Three covers advanced topics; it is aimed at anyone who might have a hand in database administration. ¬†This section covers replication, sharding, and hardware setup, in addition to indexing and query performance optimisation techniques. ¬†As a NoSQL novice building relatively small web applications, scale doesn’t feature highly on my list of priorities. ¬†However, I enjoyed learning about the problems that normalised data models face when scaling (it seems like the best you can do is stick Memcached in front of the database and upgrade your hardware), and the advantages of a denormalised data model (horizontal scaling) in this regard. ¬†I also enjoyed the section on indexing and performance optimisation; the discussion of the need for efficient indices, and how redundant indices¬†can significantly hamper query performance was particularly instructive.

Overall:

First off, I wouldn’t recommend this book to anyone without a reasonable knowledge of JavaScript, or at least reasonable proficiency in object-oriented programming. ¬†A passing familiarity with Ruby is also helpful, but not required. ¬†The application examples created by the author really help you to grasp the fundamentals, it would be a more difficult read if you couldn’t follow them.

The book is well organised – I didn’t feel the need to read parts out of order to make sense of things. ¬†Chapters Five and Six were my favourite chapters: they explain the nuts and bolts of MongoDB, and quickly get you building a real-world, practical application. ¬†They were particularly instructive on how to implement CRUD functionality, and the design patterns provided continue to be useful.

The author set out to create a book that would take a developer from being a MongoDB novice, to being a “master” of MongoDB. ¬†After reading the book, I wouldn’t exactly call myself a master,¬†but I’m certainly confident enough now to use MongoDB now in my applications, without worrying whether I might’ve been better off using MySQL.

Check out MongoDB in Action on Amazon