Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: What do you use to test Frontend code?
56 points by outside1234 on Sept 3, 2011 | hide | past | favorite | 46 comments
At our startup, we are trying to sort out the best way to handle testing against our supported Browser matrix. We want something that is repeatable and automated. We've looked into QUnit and Selenium but have this nagging feeling that we are doing it wrong.

Our application is primarily one page and uses a lot of jQuery to both animate things on screen and to work with the backend via AJAX.

Any hard won tips out there?




I always feel frustration at this question, I can't believe that sauce labs hasn't taken off. Use capybara as the driver layer, and a proper test framework that separates the different layers, and sauce labs to hit any browser matrix you want. Tie it into a CI system and never worry about this again - it's a solved problem, and it was solved by Sauce Labs.

disclaimer: I used to work there, and put my blood, sweat, and tears into building it out just for situations like this.


If you're in the Ruby world, Capybara is a great addon to Cucumber and RSpec. It solves the animation problem by adding a timeout to wait for elements in the DOM. A step like this:

When I click the link that does a 4000ms animation I should see "#mydiv"

Capybara will wait for #mydiv to appear for some timeout period (30 seconds). If it appears before then, it keeps going.

This way you can have complex animation and still assert final state of the page. Capybara allows Selenium, headless WebKit, and other JS drivers. It's one line to swap.

If you aren't in the Ruby world, you might at least steal some ideas from that lib to make your Selenium tests better.

Also, you might drop the folks at Pivotal Tracker an email. They have a fat one-page JS-heavy product, and I bet you anything they automatically test it. They might have cool stuff to share.


Capybara works pretty well for any web based front end. I'm using it to do some basic automation of websites (scheduling tasks and the like). You write the tests in ruby but honestly it is really simple: http://pastie.org/2476405 is my script for logging in to google voice and sending a text message. If I recall it was simply gem install capybara, a gem install selenium-webdriver and that was it.


Good point, I forgot that you could just use it standalone!


I'm not using full on unit tests for my current project. I just break the code into a few independently functioning layers, and make a single html file to test the code in each layer. I run the corresponding html file manually as I develop code in a given layer. It may be heresy to say this, but I've found that automated unit tests are more trouble than they're worth in a lot of cases.

The second thing I do is use my tool every day. I use the unstable branch on the unstable server whenever possible. Problems usually become obvious very quickly. I don't push to production until I've used the unstable server for 24 hours without incident. I find this combined with the testing practices described above to be the best compromise between reliability and getting-stuff-done-ability.


We use SauceLabs with Selenium, Jellyfish (very fast), Capybara (very flexible), and Jasmine. Yes, Selenium may feel hard to work with at times yet the payoff is enormous when you aim to support many browser for many years.

Edit: I have a great outsourcing team that builds Selenium for me, and runs in SauceLabs. PM me if you want details.


I use qUnit all the time because I wrote lots of jQuery plugins for everything that we do at my small startup. Then we have a QA team in India that are always clicking through the site and finding new bugs (they also have extensive selenium scripts that go through the whole site every night, but oddly enough none that run the unit tests... grr).


We use jsTestDriver at Picklive to test our game, alongside Sinon.js for a mock implementation of our game server. It's very easy to test simultaneously on multiple browsers, run tests on the command line, and we run it headless for continuous integration.

In comparison with Selenium, I've found unit and integration tests using this setup fast and resilient.


Admittedly, Selenium's standard interface leaves a lot to be desired; however, I've had great time using it in combination with a Python wrapper, where you write Selenium test cases as Python unit tests.


This. Just write the Selenium base code in the Firefox addon, and then click, export as python :D


I guess Selenium is fine, out of curiosity, why do you think using Selenium isn't the right choice for you?


I'm not sure that it isn't. I'm worried about the maintenance costs of keeping it up to date - but maybe that is the price you pay.


If you're worried about keeping it up to date, why not outsource the test runner to something like saucelabs?


Sauce is a useful service but it doesn't address the test maintenance problem... And it is a big concern.

My advice: Do NOT use the Selenium IDE. write your own framework, using Selenium, which isolates changes when they occur. If someone changes the login page, you only need to change a login method in your test suite not all 200 rest cases! Look up the Page Objects pattern as one way of doing this


I believe outside1234 is referring to the maintenance cost of keeping the tests up to date.


I'm using Selenium on a major project, and from the start refactored out common tasks. I record new tests using the Selenium IDE formerly, now the Sauce Builder plugin, then export to code and replace quite a lot with calls to our own much-simplified API (often cutting out huge sections of test code & replacing with one call).

Updating the tests even when we make significant changes to the website is must easier because of this approach.

One issue we ran into quickly... tests need to verify just one thing as much as possible, not long strings of actions that wander through the whole site. The problem is that there's considerable setup before testing that one thing -- creating users and getting them to the appropriate state, especially (worst of all when test cases require 3+ users to interact).

To address this I've set up some helper requests Selenium can make -- actions that aren't included in our production build -- to create/destroy test users of different types & in different states directly. I'm also thinking of adding a page that'll simply return a full dump of info on a given test user, so we can test that the end result of actions in the UI have resulted in the expected data.

It's all very much work in progress, but I'm convinced that with a little extra elbow grease added as pain points show up, we can keep a full suite of Selenium tests alive and useful.


For Javascript testing, I really like JSTestDriver for automation and Sinon for mocking/stubbing.

I found bulky solutions like Selenium to be somewhat hit-or-miss and overkill for most tasks. It's very rare that with proper unit testing I really need a whole additional layer just to click around the site and make sure stuff is working as expected. Maybe some basic sanity checking, but it's really not my preferred way to think about testing--which of course is just a personal preference.


Same, I feel that if there is a good coverage from unit testing, then you should only need sanity checking. Furthermore, if you are using jquery (which itself is well testing on all the browsers), then you should have more confidence that your javascript will work (providing you're not using non-cross-browser javascript on top of jquery)


For reference, the list of tools and framework mentioned on this page can be found at http://www.romku.com/list/343858/web-testing-framework. Feel free to rate them and add any missing information.


Having dealt with the Selenium and its frustrations in the past, I tend to outsource this sort of testing if at all possible. Some things can only be truly tested with eyeballs.

One tip I do have is to disable jQ animations during your test runs so you aren't having to jump through hoops in your test suite just to wait through animations -

jQuery.fx.off = true will do that.


This! Also, try to mock any setTimeouts and setIntervals so that they execute callbacks synchronously (i.e, timeout = 0).


Could try watir (http://watir.com/)


Watir is what Facebook use I think, it has plugin support for most of the major browsers and tests are fairly easy to write.


I use a custom selenium test runner that runs them in parallel (php). I plan on neatening it up and releasing it at some point if theres interest.

It also has a 'live mode' which is useful for writing up new tests. It basically follows a file and whenever there is a change it begins running the commands from that line onwards again. If it hits an exception it'll pause there and wait for you to fix the broken test/site before resuming.


I'd be interested as well.

I wrote a custom test runner as well, though not addressing the same issues; mine is specifically for non automated testing. It's a Java GUI that lets you choose from all our existing tests, tinker with parameters, and run the test live in your own browser against our test server with a few extras (like stepping through the test, go/pause, run all tests in sequence, etc.).

It's partly to let our CEO really interact with our testing -- i.e, see what's being tested, and rapidly explore the site as a whole -- but also is pretty useful as a robot; i.e., when I want to register three new users to try something out, I don't have to do it manually.


I'd be interested in seeing that. Oh, and I'd rather see a mess soon, than wait for the neated-up version.


Some things require manual testing. Animations and a lot of visual stuff needs to be analyzed holistically; there are sometimes visual quirks that are near impossible to write tests for but can easily be detected from someone quickly viewing the site.

Have different environments set up. After every big push, run through a simple user flow to make sure what the user will actually see is what you intended.


After every big push...

One way that I minimize my front-end testing liability is to do as few "big" pushes as I can. The safest change to make is the smallest change possible.


Check out Ghostbuster (https://github.com/joshbuddy/ghostbuster).


Hmmm, I'm using Splinter (http://splinter.cobrateam.info/) to do tests. It lets you automate browser actions and much more, like execute javascript, find elements returned via AJAX, interact with forms, links and more... it has drivers for chrome, firefox... You shold to test it :)


You could try with Jasmine, although it's not for this specific purpose...

And I agree with simplegeek, selenium should be fine in your case. Although can't imagine how you could test animations... :)


Jasmine isn't what you're asking for, but you should use it in addition to what you're asking for.


on a side note, do you people test graphics? I mean, it's kind of easy to say "I get a div #foo". But how do you check if it has the right shade of $color and a rounded corner and the text content is not overflowing the containing box?

I live in constant worry of changing css and totally screwing up a completely unrelated bit of the web app, and the usual thing I get told is "well, it's very fast to check if it still works", which reminds me of me not understanding why unit testing is useful.


Asking a computer to check aesthetics is start of your descent into madness. Use Selenium to check that links do what you expect, elements are loaded etc. Don't use it to check if a button graphic has the right icon loaded.

Computers are great at functional testing but you still a Mk1 eyeball to check aesthetics. Luckily, functionality is what your customers will really care about. No one will shout at you if an element is misaligned for a while before you catch it with manual testing


Side note -- I definitely agree about not using Selenium to check for alignment, images loading, etc..

But... it's pretty cool to have a single Selenium test that does nothing but zip through your site and show all of the unique views at rapid fire speed, so as soon as the page is loaded it has already filled in any relevant form and is loading the next page.

Make sure someone with good concentration sits and watches that test run once in a while; anything seriously wrong will pop out, and they can watch it zip along while taking a coffee break.


Why is unit testing useful, to you?

I have often heard two justifications for testing: (a) correctness and (b) developer speed.

Phrasing (a) differently, is X functioning as intended? Phrasing (b) differently, how quickly can I make a change to Y, and if I change Y, how long will it take to manually check Y (with a browser, etc.), how likely am I to introduce a bug (leading to extra development time) and/or will I have to rewrite a test?

If your primary justification for testing is developer speed rather than correctness, does that impact which features (models, controllers, views with div ids, actual view graphics) you should write tests for?


I'm putting extra work into unit and UI testing to build a safety net so I can convince my CEO & other developers to let me carry out serious refactorings on our code, like deleting large chunks that aren't used anymore, and changing objects that are used pervasively.

We also have an increasingly broad application; as we add new features, I (as a developer) find that I don't visit older parts of the app, though of course the users still need them to work. Particularly when refactoring basic back end stuff that's called from lots of places, it's too tedious to manually track down every part of the UI that would need to be retested due to a given change (and then test it manually... ugh).


I live in constant worry of changing css and totally screwing up a completely unrelated bit of the web app

I understand this worry, but it makes me think that it may be a design problem with the CSS. Unrelated bits shouldn't be using the same CSS, related bits should probably change together.

In a more practical note, I have created selenium tests which capture screenshots and then saved those screenshots as artifacts in the CI (TeamCity) system. Having a human look at a bunch of static images makes for a faster and less error-prone sanity test to see if things are totally broken.


I agree it may be a design problem with the CSS, I agree.

But my point is exactly that: I can't just go in and _improve the code_ (extracting classes, rewriting it in less/sass or whatever) because to verify the changes do not mess up the UI requires clicking around many pages in half a dozen browsers in three OSes.

And thank you very much for the practical note, it's an interesting idea!


You're in the catch-22 of being unable to write tests without refactoring, and unable to refactor without test coverage. I get it.

I hope the automated screenshot idea helps. I personally found it more useful in theory than in practice, but I wasn't doing a lot of UI polish work.


One other practical note, you can create a single (test) page that showcases all of the visual styles currently in use, so there is one page to check manually.


Take a look at JsTestDriver, for hooking in to your CI system.


For algorithms or application logic, even rather simple cases, unit tests are an absolute must.

For user interfaces, it's a lot more effort for a lot less return. YMMV


you might want to check out SIKULI (http://sikuli.org/) - it's like python on top of a computer vision stack, so you can write code like click([part of a screenshot]).


Selenium works fine for me.


zombiejs?




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: