-
ProTip: PV aka Pipe Viewer
May 25, 2012 by nickmulder comments
Today I was completely blown away by this insanely useful command-line tool called pv. It isn’t new and I’m definitely not the first to do a write up in a blog. I’m just stunned by the fact that I have never come across it before.
What is PV?
Its purpose is simple and clear, it measures the progress of data flying through a pipeline. It tells you how long it has been running, how fast the data is going through, it lets you know what the ETA is and best of all it shows you what percent of the data has made it through the pipe!
How to use it
In its simplest form you can use it like you would cat, the main difference being if you cat a sql file and pipe it into mysql it won’t give you a progress bar. Need to bunzip or gunzip? No problem.
Multiple Bars
You can also use it multiple times resulting in multiple progress bars. The -cN bit simply gives your bars friendly names.
How to get it
It is only a few keystrokes away.
More info on the official site: http://www.ivarch.com/programs/pv.shtml
-
Building simple JavaScript libraries with Rake-Pipeline
May 18, 2012 by rob-harper comments
At Nulayer we try to write very modular JavaScript, which allows us to be flexible and re-use those libraries in other projects. But we’ve had difficulty structuring a build process for these small- to medium-sized JavaScript library projects, while meeting the following objectives:
- Break it up: While it’s common to write smaller libraries as one big file, we like the mental separation provided by dividing the source into multiple files.
- Inline dependency management: We’d rather define each file’s dependencies right in the code than define the concatenation order in some sort of makefile.
- Keep files separate while debugging: When we’re setting breakpoints and stepping through code in the debugger we want to see our individual source files, not have to wade through one big file.
- A transparent solution: We need the production library to be free from dependency management code.
A year ago, I tried to satisfy our objectives using @jrburke’s require.js. It gave me the multiple source files with dependency management and excellent seamless development (with separate files) and production (combined and minified) environments that I was looking for. Unfortunately at the time, and things may have changed in a year, compiling requirejs out for the production build was not easy. I could get it down to a bare-bones set of require and define functions but it was still there. Requirejs is excellent for large JS projects and I ended up using it extensively for other work but it wasn’t quite the right fit for making a smaller library.
Via the Ember.js project I was introduced to livingsocal’s rake-pipeline with @wycats’ rake-pipeline-web-filters and minispade. With this combination I can prefix my source files with minispade “require” calls and satisfy objectives #1 and #2. The minispade filter for rake-pipeline can optionally encode and your source as strings, add a @sourceURL annotation and eval at run-time. In Webkit and Firefox the script debugger will show each eval’d block as a separate file, satisfying objective #3. But minispade is still required.
@wagenet’s addition of the neuter filter goes a long way to satisfying all four requirements. Neuter will determine the correct ordering of all source files and concatenate them into a single file. This is great for a production build as it means minispade is not required. So by using minispade in development and neuter in production, I can meet all four of our project organization objectives… almost.
One of the side-effects of using minispade is that all modules are evaluated within their own closure. This is great if everything that the module needs to share with other modules is exposed through a common global namespace (as in Ember.js) but it’s problematic if I want to define library-wide functions and variables that I don’t want to expose publicly. In my case, I had some helper functions that I put in a common module that was required by a number of other modules in my library. In the neutered production output everything is fine since all modules are appended as-is and can be wrapped in one big shared closure. But in development I was out of luck.
My solution was to use a modified version of the neuter filter that either outputs the raw code (in production) or outputs eval’d strings of the code with @sourceURL annotations (in development). This simple change allows me to satisfy all four of my requirements with very little work.
Here is a snippet of a rake-pipeline Assetfile that can be switched between production vs. development followed by the modified neuter filter (my changes mostly in output_source):
Using rake-pipeline with a modified neuter filter makes it easy to structure, build, and debug JS projects that use multiple source files with dependencies specified right in the code.
-
Nulayer launch night 2011!!
November 30, 2011 by peterkieltyka comments
It’s hard to believe that in 2008 we were just two guys in an office the size of a broom closet in Liberty Village. Today, two offices later (but still in the village), we are a team of more than 20 creators filled with passion and big dreams.
Since we launched Crowdreel in 2009, we’ve been flying under the radar, putting our hearts and souls into two products I couldn’t be more proud to say we made. Our best work ever: Pressly and ScoreMobile.
They are the culmination of a year’s worth of hard work by the Nulayer team. Well done everyone!
Thank you to Score Media and the Toronto Star for believing and trusting our team in the midst of many unknowns. Innovation is a leap of faith.
This leads us to November 24th, the day ScoreMobile app for iPhone hit the stores. Since the day we launched, we’ve already topped 400k users, and now sitting at #1 in Canada and #2 in US for Sports! Crazy. We have lots more planned and can’t wait to continue to innovate with the good people at The Score.
The Toronto Star’s web app, the first ever tablet experience powered by Pressly, launched a few days earlier on November 14th. So far, the most amazing result to surface is the proof we are making a difference in the way people read on the Web. Without giving up too much info, The Star increased user engagement by well over 10x. And users on all devices, from PlayBook to iPad, have been tweeting and emailing us with glowing reviews. People love the reading experience – its working!
The Star was the first step in our vision to revolutionize the way people read and discover content. As a Toronto based company, we couldn’t be more proud than to have taken that step with a great Canadian publisher.
This is just the beginning, we’re going to light it up in 2012.
Thank you to all of our friends, family and colleagues who celebrated these two major achievements with us at our Launch Night party. The next day, we roll into the office around 11am (pretty good), and Will says to me “Pete, we gotta launch more products so we can throw more parties!” … Agreed! That’s the plan.
Also, thank you to Esther Kim for planning the party, it was a hit! And thanks to Janick Laurent for the beautiful photography.
Cheers!
-
Introducing NULevelDB for iOS
November 14, 2011 by nulayer-brent comments
NULevelDB is an Objective-C API for Google’s leveldb key-value database. NULevelDB hides leveldb’s C++ interface while maintaining its functionality, using semantics and patterns which are more suited to Objective-C and more standard in Cocoa and iOS software.
We created a new Objective-C database API in response to performance issues with Core Data. Core Data is a powerful framework with a lot of incredible features. But those features do not always come for free, and they can create bottlenecks. And we didn’t need them. We needed a key-value database.
Alternatives
After evaluating a number of embeddable database solutions, leveldb looked like the best alternative. Other solutions that we considered include:
-
CouchDB is a document-oriented database sever. It’s not a pure key-value store. It lacks an Objective-C binding. CouchDB embedded still runs like a server: the database exposes a socket-based protocol interface on a dedicated thread, not a direct API. In addition, it’s quite large.
-
Berkeley DB, or “db” for short, has been around for a long time. A version of it even ships as part of iOS, but the API is inscrutable to those of us since spoiled by object-oriented design. Documentation is sorely lacking.
-
Vanilla SQLite is easy to use on its own, but, as with Core Data, it is designed to solve different kinds of problems.
We looked at other options, too, but either they weren’t mature enough, lacked Objective-C or at least C-friendly bindings, or had incompatible licenses.
leveldb
Although very new, leveldb is quite mature. It is reliable, fast, and powerful, and the API is clean and intuitive. It features concurrency support, built-in compression, and tight memory use. The fact that it’s written in C++ was only a minor impediment. Writing an Objective-C wrapper felt quite natural, even for someone with very little C++ experience.
The documentation and headers are your best guide, but here’s a quick overview.
You open a database with one function call, DB::Open(), passing a file path and options. This returns a status by value, and a new, opaque database object by reference.
Once you have a database, read, write and delete to your heart’s content. Put() take a key and a value, and adds an entry. Get()) take a key and returns a value. Delete() take a key and removes an entry. Keys and values are represented by a Slices: simple wrappers around data buffers. For efficiency, slices don’t copy data. Don’t delete their referenced data while the slice is still alive.
Besides basic operations, you can explore the database with the Iterator class. Ask the target database to create an Iterator on your behalf. Then use the Iterator to seek or step through keys sequentially. The sequence follows the sort order.
Sort order is determined by the Comparator class. You can make your own comparator, or use the default. Use it to find your boundary condition while iterating. Once you specify a comparator to go with a database, don’t change to another one—correct sorting and search of entries in the database requires that the comparison algorithm be consistent.
When you are finished with your database, delete it normally. But first, make sure you free any Iterators you have created, or it will crash.
leveldb has a a few more features, but they aren’t used by NULevelDB yet.
Getting NULevelDB
NULevelDB is available as source code from github.
The source includes an Xcode 3 project which builds an iOS universal static framework called NULevelDB. (For more about universal static frameworks for iOS, please see the iOS Universal Framework Template on github.)
The framework target depends on another target that builds leveldb as a static library. There is a third target for running a small set of unit tests.
To make the framework available to your application, build it and copy the built product to your application project directory, adding it to your application target at the same time. IN addition, you must add the libstdc++ library to your project. Also, add the -all_load option to your linker flags for all build configurations, or you will get countless C++ linking errors. If you wish, you can include NULevelDB in your project by building the leveldb library and adding the NULDBDB class files to your project directly. (You still need to make the same linking-related modifications to ensure your application builds and runs.)
Using NULevelDB
To create a new database from your application, instantiate a new NULDBDB object using the designated initializer:
or just use -init to use the default location and write buffer size.
The interface for NULevelDB mimics that of leveldb, with some simplifications and some additions. For our uses, defining separate classes for Iterators, Slices, Comparators and the rest has not yet been necessary. (These are areas of future work.) Instead, NULDBDB provides support for a variety of different kinds of keys and values, as well as convenience enumeration methods.
The simplest approach is to use keys and values that adopt the NSCoding protocol. Internally, NULDBDB converts objects to and from keyed archives and stores them as NSData. Then you only have to work with objects. By adopting NSCoding on your custom classes, they are immediately compatible with the database.
To add or update an entry:
To retrieve an entry:
To delete an entry:
In addition to reading and writing, you can check whether an entry already exists. In the implementation, this uses an Iterator to search for the provided key. If an Iterator is valid after a seek, it has found the key; otherwise the key does not exist.
In all cases, it’s up to the application to make sure that it knows what kind of object is being stored under any key, or to inspect the value to find out what it is. As is normal with NSCoding, if you use collections, objects held by them must also adopt the NSCoding protocol.
leveldb will store as much data as you ask it to, quickly and efficiently.
Faster Access for Common Cases
For better performance, use one of the streamlined interfaces. These reduce the number of times data needs to be copied. For example, there are methods customized for dealing with string keys and data values.
There are various enumeration methods, one for each of the API access styles. Most enumeration methods take two keys, for the start and end points, and a block that is called for each key found in the range. If the block returns NO, the method terminates early.
For example, you may wish to find the biggest object held by a group of keys in a certain range, all of which are prefixed by “MY_KEY:”. But you may wish to end the search early if you find a value bigger than some high water mark.
(Currently, the retain/release semantics are optional, but the enumeration APIs will probably be changed to avoid creating autoreleased objects, to reduce memory spikes while iterating.)
More Examples
The distribution includes some test projects for performance measurement. Along with the unit tests, these show the most common ways to use the database.
I have a fork of SDURLCache on github which uses NULevelDB. SDURLCache is a replacement for NSURLCache, which does not cache on disk in iOS.
There are some good opportunities for improving and enhancing the code with new functionality. Please fork us! We look forward to your pull requests.
-
