Mechanical Bee

Obj to three.js, getting your materials to show up

2012-03-21T00:00:00-07:00

We were playing around with three.js and nearly died of despair when we could not get the materials to show up on our completely black but otherwise perfect looking model for no discernible reason. In hindsight, our problem with light and not having it seems obvious but at the time, it was really not obvious.

So if someone gives you an obj file with some jpgs that are supposed to be the textures for the model and asks you to render it in your browser with three.js, here’s what to do:

Use the blender exporter or the python obj exporter to turn your obj file into a js file.
Open that js file and you will see an array of materials at the top that looks like:

"materials": [{
  "DbgColor" : 15658734,
  "DbgIndex" : 0,
  "DbgName" : "material_12.001",
  "colorAmbient" : [0, 0.0111, 0],
  "colorDiffuse" : [0.0, 0.009412000468552117, 0.0],
  "colorSpecular" : [1.0, 1.0, 1.0],
  "mapDiffuse" : "room_50k_cleaned_noroof_tex012.jpg",
  "mapDiffuseWrap" : ["repeat", "repeat"],
  "shading" : "Lambert",
  "specularCoef" : 1,
  "transparency" : 0.0,
  "vertexColors" : false
},{
  "DbgColor" : 15597568,
  "DbgIndex" : 1,
  "DbgName" : "material_13.001",
  "colorAmbient" : [0, 0.0111, 0],
  "colorDiffuse" : [0.0, 0.009412000468552117, 0.0],
  "colorSpecular" : [1.0, 1.0, 1.0],
  "mapDiffuse" : "room_50k_cleaned_noroof_tex013.jpg",
  "mapDiffuseWrap" : ["repeat", "repeat"],
  "shading" : "Lambert",
  "specularCoef" : 1,
  "transparency" : 0.0,
  "vertexColors" : false
}]

Notice how the colorAmbient values are all close to 0. Change all of these to something like [0.5, 0.5, 0.5]:

"colorAmbient" : [0.5, 0.5, 0.5]

Using the three.js boilerplate, move your js file into the js folder and the texture images into the images folder.
Open up the index.html of the boilerplate and get rid of the sample object.
Add an ambient light:

var ambient = new THREE.AmbientLight( 0xFFFFFF );
scene.add(ambient);

Load your js file with the JSONLoader and MeshFaceMaterial:

new THREE.JSONLoader().load('js/your_js_file_path_.js', function(geometry){
  var mesh = new THREE.Mesh( geometry, new THREE.MeshFaceMaterial());
  scene.add( mesh );
}, 'images');

Finally start the local server and navigate to localhost:8000 in your browser.

make server

Organizing a large backbone and rails app, some thoughts

2011-12-17T00:00:00-08:00

We recently started working on a new web app with backbone 0.5.3, rails 3.1, and couchdb. It’s not the still-raw-and-pulsing bleeding edge but it’s pretty new stuff. There are some nice examples of backbone apps, e.g. DocumentCloud, CloudApp, CloudEdit, Todo, etc., but conventions and best practices aren’t there yet in the way that they are for Rails.

Our app is larger than any of these examples so we thought we’d share how we’re currently organizing it, and handling routing, forms, and nested models.

Folders and Files

Our app files and directory structure are very similar to that of backbone-rails with a few differences. backbone-rails creates four folders under app/assets/javascripts/backbone:

routers/
models/
templates/
views/

Our app is in stealth mode right now so let’s say it’s a web app to connect Jarls, Guild Masters, and other quest givers with adventurers who want to do quests. Quest givers and quest takers have similar but different UI requirements. For example, they both look at quests but quest givers can edit the quest and see all the quest takers who are interested in the quest, etc. So instead of one backbone folder for the entire app, we’ve got folders for both groups and a folder for shared assets:

app/assets/javascripts/quest_givers 
app/assets/javascripts/quest_takers
app/assets/javascripts/shared

Routing

There are two types of routes the backbone app needs to know about.

backend rails routes so that models can sync with the server
front-end routes for navigating around, triggering views and actions, etc

For the backend routes, we could just set it on the Collection e.g.

class QuestGiver.Collections.QuestCollection extends Backbone.Collection
  url: '/quest'

Instead we added an application.js.erb file to our rails app and filled a Skyrim.Routes object using the rails helpers. Something like:

var Skyrim.Routes = {}
Skyrim.Routes['quest_givers_quests_path'] = '<%= quest_givers_quests_path %>'

Then our Collection url is:

url: Skyrim.Routes.quest_givers_quests_path

Front-end routes are used in a couple places:

in the backbone routers, typically:

class QuestGivers.Routers.QuestsRouter extends Backbone.Router
  routes: 
    "quests" : "index"
    "quests/new" : "newQuest"
    "quests/:id" : "show"

in the backbone views for example when switching from new to show after a model is saved:

class QuestGiver.Views.Quests.NewView extends Backbone.Views
  save: (e) ->
    @collection.create(@model.toJSON(),
      success: (quest) =>
        @model = quest
        window.location.hash = "/#{@model.id}"
    )

in the jst templates in places like the href of anchor tags e.g. :

<a href="#/<%= id %>">Show Quest</a>

However we wanted our routes defined in one place so we added a Routes object:

window.QuestGivers.Routes =
  quests: '/quests'
  new_quest: '/quests/new'
  quest: '/quests/:id'

Then we used a slightly modified version of router.js and pass QuestGivers.Routes to addRoutes(). Now our routers, views, and templates look like:

class QuestGivers.Routers.QuestsRouter extends Backbone.Router
  initialize: (options) ->
    @routes = {}
    @routes[QuestGivers.Routes.quests] = 'index'
    @routes[QuestGivers.Routes.new_quest] = 'newQuest'
    @routes[QuestGivers.Routes.quest] = 'show'
    @_bindRoutes()

Don’t forget to _bindRoutes() when adding routes this way or they won’t get triggered.

class QuestGiver.Views.Quests.NewView extends Backbone.Views
  save: (e) ->
    @collection.create(@model.toJSON(),
      success: (quest) =>
        @model = quest
        window.location.hash = QuestGivers.Routes.quest(@model.id)
    )

<a href="<%= QuestGivers.Routes.quest(id) %>">Show Quest</a>

Forms and Nested Models

We’re not using the backbone-rails gem but we are using the datalink plugin to connect models with form fields and the rails-ified Backbone.sync. We’re also using backbone-deep-model so that we can do:

class QuestGiver.Models.Quest extends Backbone.DeepModel
  defaults: 
    name: null
    address: new Skyrim.Models.Address()
    
class Skyrim.Models.Address extends Backbone.Model
  defaults:
    address: null
    city: null
    state: null
    zipcode: null

model = new QuestGiver.Models.Quest()
model.get("address.city")

Going Forward

We’re just getting started so maybe in a few months we’ll look back on this and shake our heads in wonder. We’ll see where it goes!

Why I'll (probably) never use CoffeeScript

2011-10-12T00:00:00-07:00

We started a new project using Rails 3.1, so I was checking out all the new stuff in this release. Most of it looks great, since the new baked in features solve problems that we were using third party or home grown solutions for anyway. On our last project (Rails 3.0), we used Less and Jammit. On this one, we use Sass and Sprockets. Rails chose for us and that’s great, because now we don’t have to deal with it. But there’s one new feature that I just don’t get: CoffeeScript.

I mean, I get it. Javascript has a lot of pomp and isn’t a whole lot of fun to code in. But it’s the only language that the browser runs and that’s a real constraint. If I write my client-side code in CoffeeScript, the browser still receives and runs Javascript. That creates a huge disconnect between what I’m writing and what I’m running. Now maybe your code runs perfectly every time, but mine doesn’t. And when it doesn’t, I look at what’s happening in the debugger. What do I see in the debugger if I use CoffeeScript? Not my code, but a derivative of my code. What do I have to do to fix a bug? Mentally decompile the running Javascript to the source CoffeeScript, make the change in the CoffeeScript, and hope that it compiles back down to the Javascript that fixes my problem. Harsh.

The problem here is that CoffeeScript fails at it’s fundamental purpose as an abstraction: removing me from the underlying complexity. It in no way removes me from having to deeply understand Javascript. In fact, it increases complexity because now I have to know Javascript and CoffeeScript and be able to mentally translate between the two.

Now, as forward thinking person, I do see the value in alternate client side languages. I suspect that if lots of devs start using CoffeeScript or Dart or whatever, the browsers will eventually implement some kind of native support for these languages, which will be great. But this is a long way off and probably requires one language to dominate and all browsers to support it. I’m just not willing to let my own code suffer for years helping this to happen. Maybe you are, and for that I appreciate your sacrifice.

You can check out our new CoffeeScript-free project at wheretoeatat.com

Geek fun: building a lil' node.js app

2011-07-28T00:00:00-07:00

We recently wrote a little service in Node.js and deployed it on Heroku’s Cedar stack. All it does is return UPS shipping rates when we do an http get with certain parameters. So instead of hitting UPS directly every time an item page loads, we hit the node app and it either pulls the rate out of Redis or asks UPS, meaning that shipping rates should come up a lot faster. We didn’t really need to spin up a Node app for this but hey, this is how we have fun :D.

It’s a pretty basic app but there were a couple modules we used for the various pieces:

Query string parsing - qs
Caching - RedisToGo and redis
XML templating and parsing - Mu and xml2js

And as an auxiliary bonus, anyone can use it to get UPS public rates.

Node modules

qs

Heroku runs Node v0.4.7 and there’s an included query string module that parses the query string into an object. But we wanted to send nested parameters like ‘destination[state]=NJ&destination[zip]=07305’. qs adds back the parsing of nested parameters that apparently got removed in 0.3.x. For example, hit http://localhost:8080/?destination[state]=NJ&destination[zip]=07305 in your browser while running this with node locally:

var http = require('http'),
    qs = require('qs'),
    url = require('url');
http.createServer(function (request, response) {
  var query = url.parse(request.url).query;
  response.end(JSON.stringify(qs.parse(query)));
}).listen(8080);

redis

We don’t charge flat shipping rates, it’s just whatever UPS says it is and it’s based on a number of factors. But since there are only 7 zones in the continental US and most things on Givmo are within 1-15 pounds, there’s a small set of probably base rates, ripe for caching.

We decided to use redis partly because it seemed like the node community preferred it and partly because we thought it’d be fun. So we added the Nano plan for RedisToGo on Heroku and node_redis, hooking them up with this little snippet. Each key is set to expire after 24 hrs.

mustache and xml2js

The XML going back and forth is one of the annoying aspects of the UPS api. For the request, we made an XML file with mustache template tags. A snippet:

<ShipTo>
  <Address>
    {{#destination}}
    <PostalCode>{{zip}}</PostalCode>
    <StateProvinceCode>{{state}}</StateProvinceCode>
    {{/destination}}
    {{#residential}}<ResidentialAddress />{{/residential}}
  </Address>
</ShipTo>

Then mu to compile and render the template. When UPS returned an XML response, we used xml2js because it was simple and we liked how it mapped XML to a js object.

GET public rates

Anyone can use this to get UPS public rates. Why would you want to use it instead of the real UPS api though?

Faster (except when AWS goes down and takes out Heroku/the internet).
Easier: No need to get a key or deal with xml.
Just for fun :)

Here are some ways to use it:

From your browser

With Node.js:

var http = require("http"),
  qs = require("qs");

var query = qs.stringify({
  origin: { state: 'NC', zip: '27713' },
  destination: {state: 'NY', zip: '10001' },
  weight: '20',
  residential: 'true'
});

var options = {
  host: 'hollow-spring-510.herokuapp.com',
  path: '/?' + query
};

http.get(options, function(res){
  res.setEncoding('utf8');
  res.on('data', function(chunk){
    console.log(chunk);
  });
}).on('error', function(e){
  console.log("Error: " + e.message);
});

With Ruby:

require 'rest_client'

params = RestClient::Payload.generate(
  :origin => {:state => 'NC', :zip => '27713'},
  :destination => {:state => 'NY', :zip => '11211'},
  :residential => true,
  :weight => 2
)

puts RestClient.get "http://hollow-spring-510.herokuapp.com/?#{params}"

couch_record: a new CouchDB ORM for Ruby on Rails

2011-07-13T00:00:00-07:00

We’ll write a more detailed post about it later but we’re excited to open this up now: couch_record, a new couchDB ORM for Ruby on Rails, using couchrest and based on our experiences with couchrest_model and, to a lesser extent, couchrest_rails.

Why do we need another CouchDB ORM?

When we were deciding which ORM to use, it seemed like couchrest_model had the most traction and most activity. But there were a number of aspects we were unhappy with (we wrote about one of them here).

Then recently we added New Relic and noticed some really problematic performance issues with our items page (to the point where we were restarting our app every 20 minutes to keep response times reasonable). We pinpointed the issue to a memory leak from couchrest_model. We were a couple versions behind and the latest version fixed our problem but also introduced a number of new things that didn’t work with the rest of our app. So we decided to roll our own with these goals:

Be Fast - Your app has to map a lot of objects, so try to do as little as possible, especially when reading records from CouchDB
Be Simple - You’re going to end up looking at (debugging into) your ORM when something’s not working as you expect, so the less code there is and the easier it is to understand, the better.
ActiveModel - Use ActiveModel modules wherever possible to support ActiveModel functionality
Don’t Support Everything - CouchRecord is not a drop in replacement for ActiveRecord, CouchRest::Model or anything else

We made this for ourselves so it very specifically addresses our needs and the tests and documentation are a bit oblique but we’d love for other people to try it out and find it useful in their own projects!

Why we don't use the Heroku SendGrid addon

2011-07-10T00:00:00-07:00

TL;DR: When you use Heroku’s SendGrid add-on, your account is a sub-user of the main Heroku account and doesn’t get manually provisioned or tiered based on goodness like solo SendGrid accounts. This means that the IP group your in is more likely to be populated with miscreants whose bad behavior decreases your own deliverability. So if you’re not a miscreant and don’t want to get lumped in with other miscreants, you should just sign up for a normal SendGrid account—the pricing/plans are the same or better anyways.

Using SendGrid via the Heroku addon is brain-dead easy

Using a normal SendGrid account with a Rails app is easy—sign up for an account and modify the smtp config for action mailer. You know what’s easier though? Clicking a button. And that’s how the Heroku SendGrid add-on works: click a button and SendGrid starts delivering your app’s emails. With no discernible difference in features or price, we picked the easiest option and activated the Heroku SendGrid add-on. Ticket complete.

But then people started complaining that they hadn’t received their confirmation email or password reset email in 6, 12, 24 hours. What was going on?

At this point, we have to pause to commend SendGrid support. We were at a Hackathon for New York Internet Week and mentioned to a SendGrid engineer that we had noticed some problems with deliverability. He said he’d inquire with his compatriots and honestly we didn’t really believe him since a) we said it in a joking tone, b) they were visiting New York from Cali and who wants to remember complaints from randoms on another coast, and c) we are tiny fish in their vast ocean, paying exactly nothing. But he did and they were so responsive and solicitous, we were astonished they could get any work done in between their many detailed, specific emails.

Sharing with sketchballs makes you look sketchy

So what was going on with our emails? We signed in to Sendgrid with the account credentials Heroku had created for us (use heroku config to get your username and password) and under the email activity tab, found that a number of emails had been “deferred” with these errors:

reason: 452 4.2.2 The email account that you tried to reach is over quota. Please direct the recipient to http://mail.google.com/support/bin/answer.py?answer=6558 z5si15616573yhh.117 (throttled)

reason: 450 4.2.1 The user you are trying to contact is receiving mail at a rate that prevents additional messages from being delivered. Please resend your message at a later time. If the user is able to receive mail at that time, your message will be delivered. For more information, please visit http://mail.google.com/support/bin/answer.py?answer=6592 w4si8027019anc.83 (throttled)

reason: 421 4.7.0 [TS01] Messages from 67.228.50.54 temporarily deferred due to user complaints - 4.16.55.1; see http://postmaster.yahoo.com/421-ts01.html (throttled)

reason: 421 4.7.0 [TS01] Messages from 67.228.50.55 temporarily deferred due to user complaints - 4.16.55.1; see http://postmaster.yahoo.com/421-ts01.html (throttled)</pre>

From the help links in the first two errors, it looks like the problem is with the recipient. But based on the latter two errors, it looks like the problem is with emails coming from the ip addresses. Everyone using the Heroku SendGrid addon gets added as a subuser to the master Heroku account and shares an IP group. This means that if other subusers are up to no good, efforts to curb their sketchy email will affect our legitimate email as well. The obvious solution is to switch to one of the SendGrid plans that include a dedicated IP address but we preferred the price of our current plan (free).

If you must share, pick a good neighborhood

What we discovered: all normal SendGrid accounts get manually provisioned and those without dedicated IPs get sorted into tiers based on goodness. Accounts that get added through Heroku, since they are subusers of the Heroku account, don’t go through either of these filters.

From our favorite SendGrid engineer:

…Our Bronze and Free packages are generally a bit better than Heroku. One reason for this is; Free or Bronze accounts are tiered IP groups based on reputation, so if you guys are not bouncing a lot of emails and don’t have many complaints you will filter into the highest tier of IPs which will have better delivery rates than the lower tiers. I believe we have four tiers total, so that gives some better granularity to pair better customers with better customers rather than lumping you all together.

This tiering doesn’t happen on Heroku accounts, so everyone is all together, good, bad, or ugly…

…the fact that there is an increased complaint rate on the Heroku IP group means someone is not behaving themselves and is probably sending out either something resembling spam, or to a list that they didn’t really opt in. Either way, this bad for all the rest of the upstanding citizens on the Heroku IP Group. So, we will need to look into this and see if we can find an eliminate the guilty party. Heroku accounts unfortunately are a bit more prone to this, as the accounts are subject to Heroku on-boarding procedures, and thus get to skip the SendGrid manual provisioning process…

Heroku is great for many reasons, one of them being the addons ecosystem. Just don’t use the Heroku SendGrid add-on to add SendGrid to your Rails app. We’ve discovered that it’s well worth the extra 10 minutes required to sign up for your own account; we’ve since switched to a normal SendGrid account and have noticed significantly fewer problems.

How to auto reload gems that you're working on in Rails 3

2011-06-28T00:00:00-07:00

We recently moved our CouchDB ORM code into its own Ruby gem because we’re thinking about making it open source. While working on the new ORM gem in the context of the Givmo Rails app, I wanted the the gem code to reload on every request like the rest of the code for the app. For a while I was making frequent changes and restarting the server every time got old pretty fast.

I eventually figured out how to do this, but it wasn’t super easy to find, so here’s the short, sweet version of what you’ll need to do:

First in development.rb, you need to add the lib directory for your gem to the autoload path. This is because gems normally only get loaded once, on startup. Since we want it to reload, we use the Rails autoload system. Files only autoload if the relevant constant is missing, so the last step is to tell rails to unload the root constant for our gem on every request.

Then in your Gemfile, tell bundler to load the gem in question from wherever you’re working on it. In this case, it’s in a sister directory to the Rails app.

How to find your SendGrid login when using the Heroku addon

2011-06-12T00:00:00-07:00

If you’re using the Heroku SendGrid addon, they’ll create account credentials for you. e.g. a username and password. Sometimes you want to either do some configuration or more frequently, log in to the SendGrid site to see stats and activity and stuff.

heroku config

and you’ll see a bunch of config vars. The ones you’re looking for are:

SENDGRID_USERNAME
SENDGRID_PASSWORD
SENDGRID_DOMAIN

Omniauth, openid, heroku, and https

2011-05-11T00:00:00-07:00

We recently added the option to log in or sign up with Facebook, Twitter, or Google using the handy omniauth gem.

Facebook and Twitter went swimmingly. We had already been integrating with Facebook and Twitter so that people could tweet or post to Facebook to tell their friends about the items they’re giving away. Adding a sign up/log in pathway was just an extension of that functionality.

But we ran into a problem with Google and OpenID: they worked fine in development but kept failing with “invalid credentials” in production.

There are two main differences with our production environment:

It’s on Heroku which has a read-only system
It always runs over https

We looked at possible problems with writing to the filesystem on Heroku first. Even though we had the filesystem path set to ./tmp, we were still worried about it and thought about switching to a memcache store. But then we tested another app on Heroku with the same filesystem store writing to ./tmp and that worked fine.

Okay maybe it was a problem with ssl. This turned out to be really hard to google because we kept trying some combination of “Omniauth openid heroku https” (thus the blatant google mongering of this title) but in fact the problem was with Rack. We’re using rails 3.0.3, omniauth 0.2.5, devise 1.3.4, and rack 1.2.2. The 1.3.0.beta version of rack has a notably different request.rb from the 1.2.2 version. Notice the scheme and port methods.

request.rb in rack 1.2.2:

def scheme;          @env["rack.url_scheme"]                  end
def port;            @env["SERVER_PORT"].to_i                 end

request.rb in rack 1.3.0.beta:

def scheme
  if @env['HTTPS'] == 'on'
    'https'
  elsif @env['HTTP_X_FORWARDED_SSL'] == 'on'
    'https'
  elsif @env['HTTP_X_FORWARDED_PROTO']
    @env['HTTP_X_FORWARDED_PROTO'].split(',')[0]
  else
    @env["rack.url_scheme"]
  end
end

def port
  if port = host_with_port.split(/:/)[1]
    port.to_i
  elsif port = @env['HTTP_X_FORWARDED_PORT']
    port.to_i
  elsif ssl?
    443
  elsif @env.has_key?("HTTP_X_FORWARDED_HOST")
    80
  else
    @env["SERVER_PORT"].to_i
  end
end

When we go to /auth/google in production, rack 1.2.2 doesn’t realize we’re coming from https and the return_to for OpenID gets set to http. In development, this was fine because everything is http but in production, OpenID does a verification on the return_to url and raises a ProtocolError when the schemes don’t match.

idres.rb:199:

[:scheme, :host, :port, :path].each do |meth|
  if msg_return_to.send(meth) != app_parsed.send(meth)
    raise ProtocolError, "return_to #{meth.to_s} does not match"
  end
end

Then the OpenID strategy in omniauth fails with the not so useful “invalid credentials” message.

open_id.rb:91

@openid_response = env.delete('rack.openid.response')
if @openid_response && @openid_response.status == :success
  super
else
  fail!(:invalid_credentials)
end

We ended up monkeypatching it with this code and it seems to be working so far.

Performance issues with CouchRest::Model and ActiveModel

2011-05-03T00:00:00-07:00

For Givmo, we have an admin page that lists about 100 records at a time and it was taking upwards of 10 seconds to load. It doesn’t do anything super complicated, so I didn’t really have a good starting point for what to optimize. Time to fire up the profiler.

The page in question loads 100 records (documents) from a CouchDB database and displays them to the user in a table. Nothin’ crazy. We use CouchRest Model to do our ORM, which makes our model classes behave like ActiveModels. This makes them more or less compatible with tools that like to work with ActiveRecord.

I wrapped the offending code with the ruby-prof gem and generated some profiling data. After some digging, I found that all the time was being spent in 2 methods:

CouchRest::Model doesn’t create accessors for the model’s properties. Instead, it keeps a list of all the properties (if you’re looking, it’s in couchrest_model/lib/couchrest/model/properties.rb) and lets activemodel’s #method_missing deal with it (activemodel/lib/active_model/attribute_methods.rb). So when a record is being loaded from the database and you have a property called ‘email’, it first checks #respond_to? to see if there’s a setter (#email=) and then calls the setter which triggers #method_missing.

Both #respond_to? and #method_missing have to do roughly the same thing: check if there’s a property named ‘email’ in the list of properties. They both call the same method to do this: #match_attribute_method?. This method (and its children) could use some optimization. It maps the array of Properties to an array of property names (creating a new array of names every call) and then loops through the new array looking for the name. For 100 records with 10 properties, this gets called about 20,000 times. Ouch.

So, for this particular page, I cut CouchRest::Model out of the picture. The time to render the page went from 10 seconds to 0.077 seconds. That’s 2 orders of magnitude folks.

Incidentally, the query to get the data from the Cloudant server, over the internet, though my measly cable modem takes about 0.5 seconds. Something doesn’t feel right with it taking 20 times longer to map the data into objects than it does to load it from the database.

Oh and we’re on Rails 3.0.3 and couchrest_model 1.0.0.beta8.