You are your target market

Eat your own dog food. It goes for writers and coders alike. Write things you want to read. Make tools you will use.

We tend to think our experiences are unique. They’re not. The inconveniences you put up with are inconveniencing others. The information you can’t find is eluding others. The answers you don’t have are being sought by someone besides you.

But there’s a subtle split in approach for writers and developers, and that subtelty reveals one of the biggest philosophical differences I’ve found between traditional reporting and the web work I’ve evolved into. When you write stories, you  universalize. No self-respecting reporter would pitch a story about a speed trap where they themselves got caught. But on the web, universalization is the opposite of personalization and individualized service. Any developer worth her salt knows that speed trap experience calls for a solution (Bad example of course, since several already exist).

This shift in thinking has had a huge effect, personally. Curbwise (which, yes, I’ll probably be talking about for awhile) started out as an amorphous idea, but became more targeted as I navigated the treacherous home-ownership waters. My experiences informed the UI, the data flow, the information and how it was presented. The shift has been made. Now, I can’t stop. Each question I can’t answer is an app. Each answer I have to dig for is a database. My list of ideas has tripled. The number of proof-of-concepts sitting on our dev server has grown.

Online, a self-centered approach is no longer a nasty trait, but a bolt of insight to leverage.

Making lemonade

Earlier this week, Omaha got knocked out of the running in Code for America. No two ways around it: that sucks. I wish we could have watched that process unfold, and I was hoping to learn more about the idea the city had submitted.

Now we have a choice to make. We can either let the idea die on the vine, or not. As easy as it is to turn the other cheek, let me make a humble appeal.

It doesn’t take a great diviner to see that Omaha has an impressive, hungry, talented creative community. You can read about it on Silicon Prairie. You can feel it at Big Omaha, at Hacks/Hackers, at Open Coffee or any of the other things we do just to do. There are people making products and building things. They care about this place, and want to see it improve.

There’s also this whole open source thing. Yes, that means free software. But it’s also a philosophy, something difficult for people who don’t code to grasp. It means collaboration. It means working on things because you think they’re important; it means contributing to a project because you think it’s fun.

So, city leaders, I say we give it a go. Invite the coders to come down and hear your idea. Have a conversation. Share. I think you’ll be surprised to learn how invested and capable your city’s creatives can be. And your creatives, well, I think they’ll be happy to know that your ends are kind of the same as their own. With a little bit from both sides — the oilcan of political will from government, a little bit of elbow grease from the Makers — I think we can pull something off.

We’ll probably impress ourselves. Hell, we might even impress Code for America.

Google Mercator and PostGIS

One error that tripped me up for a stupid amount of time while trying to wrap my head around GeoDjango/OpenLayers was projections. I get, in an academic way, what they are and why they’re important, but I still sit slack-jawed and stupefied when something goes wrong.

It took a lot of searching around to find out that PostGIS doesn’t ship with the reference system to overlay maps on Google tiles. To make that happen, you have to define the projection. You can do that by adding it to yer spatial_ref_sys table, like so:

INSERT into spatial_ref_sys (srid, auth_name, auth_srid, proj4text, srtext) values ( 900913, ’spatialreference.org’, 6, ‘+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs’, ‘PROJCS["unnamed",GEOGCS["unnamed ellipse",DATUM["unknown",SPHEROID["unnamed",6378137,0]],PRIMEM["Greenwich",0]
,UNIT["degree",0.0174532925199433]],PROJECTION["Mercator_2SP"],PARAMETER["standard
_parallel_1",0],PARAMETER["central_meridian",0],PARAMETER["false_easting",0],PARAMETER
["false_northing",0],UNIT["Meter",1],EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"]]’);

More on Curbwise

Yesterday, we launched Curbwise. I thought I’d prattle on for a bit about things I learned. This is more 10,000-foot-view stuff. Hopefully I’ll get to brass tacks in the next few days.

Focus on what you know. The site is built on GeoDjango with PostGIS on the backend. I have no idea how to use postgre, much less PostGIS. But it doesn’t matter, because I know my tools. Using Django, there is nothing that can be done in a database that can’t be done with Python. Tables can be created, indexes can be added, relationships can be defined, data can be imported and smushed around. Ben Welsh taught me this in one of our numerous counseling sessions (Thanks dude), and after about the 15th time it stuck. We had an aggressive schedule for this site, so if a skill wasn’t absolutely, completely, utterly necessary, I skipped it. This leads to a hacky approach that flies in the face of everything I know about ‘using the right tool for the job.’ But you know what? Proof’s in the pudding. Site runs and is exceeding my expectations. Everything went better than expected.

GeoDjango is powerful… and that can be scary. This is probably just my paranoia, as smarter people have told me it’s not worth worrying about. But. GeoDjango is as resource-intensive as it is cool. We went through several iterations, with one major refactor reserved for ditching geoqueries that weren’t absolutely necessary.  GeoDjango can do some amazing things. In this case, it allows us to paint a detailed, granular story on the real estate market for every house in the county. But doing that is costly, and so we wanted to find workarounds. That includes, for instance, hardcoding subdivisions into the data rather than running a superfluous “contains” query. GeoDjango is a great tool, but in a production setting, we chose to use it sparingly.

The open-source mapping community is astonishing. And you should start poking around in it. Right now. For this project, we used OpenLayers, which allowed us to build the uber-cool clicky maps available on pages like this property profile.  We did some other fairly off-the-shelf stuff, like adding out own icons and some futzing with the layerswitcher. But we barely scratched the surface. Mapping is an area that is set to blossom. Geolocation is becoming a basic contextual point for mobile and, with HTML5, even the desktop. At the same time, increasingly sophisticated mapping tools are becoming easier to use and easier to customize. Here are some neat  projects that got used or looked at or otherwise piqued my interest. Go look now.

  • Tilemill – Allows you to design maps. Like you would design anything else, from scratch.
  • OpenLayers – Javascript mapping.
  • Mapnik – Making map applications.
  • QGIS – Open source ArcView, basically. Better, IMHO.
  • HTML geolocation – just a demo of what’s to come.

Play to strengths. I know how to code a site. That was not enough. Paul Goodsell has been covering tax assessments in Douglas County since I was a kid. I’d go out on a limb and say he knows the system better than the assessor, who has held the job for only a portion of Paul’s career. He pulled together the data and made it work for the models I built. He also cooked up the algorithm we used for the custom reports — something I wouldn’t have tried on my own. Ben Vankat, meanwhile, is a jack-of-all-trades who has become a wizard with javascript and the django template system. The clean separation of skills allowed us to get this done in an ambitious time frame. It’s also a credit to our framework how little overlap there was between the three skillsets.

 

Thinking business

Today, we launched Curbwise, the most ambitious application I’ve worked on. It focuses on property sales and assessments in Douglas County, an area that covers the largest part of our metro area readership. On the surface level, there’s a lot I’m proud of:

  • We’re breaking news to every homeowner in the county, showing them for the first time how their tax assessment changed from year to year.

  • We’re allowing the user to quickly see the valuations for their immediate neighbors, and how they changed. They can also look at recent sales and valuation per square foot comparisons within a quarter mile.

  • We’re giving users an easy way to stay on top of sales in their neighborhood, providing RSS feeds and easy-to-use maps. On the main page, we’re also highlighting the area of town with the most sales in the past 30 days and comparing the recent week to the seasonal trends.

In short, it’s a property site aimed at homeowners. Not prospective homeowners. Not people looking to move. This is for people who actually own homes and want to be informed about their highly-local market. Had we stopped there, I’d be happy.

As the title implies, we didn’t.

Instead, we tried something. We added a transactional component. Users can elect to have us run an algorithm that analyzes their home’s valuation in comparison to other, similar homes in a nearby area. The results can be used in a valuation protest where homeowners try to get their taxes reduced by arguing that their home is overvalued. Users can purchase the report for $20 through a PayPal process that, on its own, might necessitate a how-to post.

The idea was planted in my head by Steve Doig back when I was in Phoenix. He did something similar in his days at the Miami Herald. As he explained in an e-mail:

“… I wrote a SAS script that would produce a printout by subdivision of all the assessed values of all the residential parcels. Readers could send us $10 and get a report of the assessments in their area mailed to them, for possible use in challenging the assessment. I came up with this after having done a sales/assessment ratio study the year before that showed the assessor’s office was under-assessing expensive homes; i.e., a tax break for the rich. I’m sure it was coincidence <cough, cough> but the next year my own house assessment went up far more than that of any of my neighbors. My motto is “don’t get mad, get even” so I created the printout and thereby prompted hundreds of appeals.”

I’ve written before about the need for newsroom developers to start experimenting with the revenue model. We’re pulling apart the old ideas of stories and narratives at the foundation, redefining how people experience journalism. I argue we can bring that same energy to making money. This is me putting my (other people’s) money where my (the World-Herald’s) mouth is. I think we’ve come up with a good solution. We’ve picked a tool with a niche audience and real monetary value. We’ve separated basic site behavior from higher-level functions.

We’ve filtered the basic from the advanced. We’re not selling our data. We’re selling an analysis of our data. We’re selling our expertise, which is precisely what we should be looking to capitalize on.

I can’t give specifics on how this project fares, but I will be as candid as I can about how we fare in terms of uniques and views over the next few weeks.

Update 1: Day 1 of the site was a smashing success. We had a time on-site of around ten minutes and, before 5 o’clock, we sold more custom reports than I thought we would sell in the first week. Day 1 also revealed an interesting trend. With most news apps I’ve worked on, your traffic sweet-spot is either early in the morning or in late afternoon — basically when people are screwing off at work. If you haven’t peaked by 5, you’re not going to peak. Curbwise didn’t fit that model. Traffic *doubled* after 5. Doubled. As in multiplied by two. What makes that more exciting is that I only know the number of reports we sold before 5. I hope that conversion rates jumped once people got into their own homes.

 

 

 

 

“Data is the pollution of the information age”

says Bruce Schneier, founder of BT Counterpane.

Django trick: Group by datetime

Useful, hard to find. And so I save. http://www.jpichon.net/blog/2010/5/django-group-by-datetime/

Learning how to learn

It hits you. The idea. The one that’s going to change the world, make your millions. You have audience, and a business plan, and you know exactly how this thing is going to work. You sketch mockups. Brainstorm catchy names. You buy the url. All that’s left? Building the damn thing.

For people who don’t know code, that’s where things get tricky.

Building in code is trial and error. It’s highs and lows, sprints and unending waits as you puzzle out that *one* issue. And that’s after you already sort of know what you’re doing. Getting to that point can be a trick. There are books. Classes. Various walkthroughs and tutorials. Everyone learns differently, and all have merit. But learning to code boils down to one thing:

Just start.

That’s it. Do it. Dive in and make something. It’s the only option. You’re not going to learn to swing a hammer by reading about it. You have to take a few swings, bash your thumb a bit, and eventually figure out what you’re doing.

Learning to code is more or less the same. If you actually want to learn to program, this advice doesn’t sound terribly helpful. If you try it, you’re going to fail quickly. And often. But here’s a secret: programming is failure. Repeated failures with small progressions. That idea in your head, of the guy coding something up, then running it and having it work perfectly? Banish it from your mind. Any product is the end result of umpteen failures large and small. But in the end, you’ve solved a hundred problems you’ll never have to solve again. And your thing? Done. It’s a great feeling.

If you’re trying to learn to develop websites, here are a few ways to get started and keep yourself progressing.

  • Cheat. For me, one of the biggest obstacles to learning Python/Django was getting to the point where I could start. Installation isn’t hard, but it’s tricky enough to scare off people with no idea what they’re doing. (Hi. That was me). If you’re toying around locally, there are a handful of easy-to-use installers that can get you up and running quickly. BitNami is one example I’ve plugged before and still favor. If you want to use a live server, Webfaction makes things pretty stupidly simple, and their support is tops.
  • Borrow. Just about every respectable language and framework has a decent tutorial to introduce the basic concepts. Django has a four-part walkthrough. Ruby on Rails, another framework, appears to have a fewsimilar options. And there are other options. The beauty of working in the open-source world is that everyone shares. Code repositories like Github are full of functional projects you can hook into to see how other people get things done. (If you use any of their work, though, make sure to follow the rules of their license.) There are also good people who, out of the goodness of their heart, share tricks and tips. If you want to find the Python jesus, you could do worse than Ben Welsh’s “recipes” for accessible, well-explained code.
  • Steal. Chances are, you aren’t the first person to run into whatever problem you’re running into. Use that to your advantage. If you get an error , paste it into a search engine. If you don’t, hit Stack Overflow. See if there’s a support group in Google Groups. See how other people have solved your problem. And if that doesn’t work…
  • Beg. To paraphrase Marcellus Wallace, pride is overrated. You’re going to get stuck. You’re going to have a stupid question. You’re going to come upon a concept your head can’t envelop. Don’t be afraid to ask someone for help. The worst they can do is call you names. They’re not going to.
  • If all else fails, study. The documentation for just about any language is pretty good. The only reason I put it last is because, at least for me, it didn’t do my much good as I was feeling my way around larger concepts.

These tips are far, FAR from the only way to learn. They also might not represent the best approach. If I’m missing any other favorite tricks, I’d love to hear them.

A final note. This post has been sitting in my drafts, in one form or another, for a while. Why? I still don’t consider myself a programmer, much less in a position to tell others how to get to where I am. But I get asked some version of the “How do I start?” question at least once a week, and figure it’s time to cut through some of this “mysticism of the web geek” crap.



 

Ha

A Google Image search for “Caspio” yields my CaspioFail. For old time’s sake:

Putting the admin to use

We have a working model and an admin that allows us to interface with its data. It’s functional and looks pretty, but it’s still not terribly helpful. Let’s soldier forth.

Managing Users and Permissions- First thing first. You’ll notice you have a built-in app (providing django.contrib.auth was installed, as it should be) that includes two editable models: groups and users. This is, amazingly, where you can control groups and users.

It should be pretty intuitive. Dive into the “users” model and add a test account. Save that and you’ll be taken to a second page where you see how customizable each account can be. Here’s a quck rundown of what everything does:

  • The user’s first name, last name and e-mail address are not required, but can be nice to have on hand.
  • In order to log in, a person has to be both staff and active, so both those boxes have to be checked.
  • Superuser status gives a permission full permission to do everything. They can add and edit anything that shows up in any app. Be careful in giving someone this level of access.
  • Finally, we reach  “User permissions.” This is the meat of any CRUD interface. Each model has three different levels of access that a user can have: Add, Change and Delete. They do exactly what you think they should do. Any user with Add permission can add entries to the model. Any user with Change permission cannot add a new listing, but can make changes to any existing entry. A user with Delete permission can’t add or change anything, but can zap anything from the system.
    Obviously, it doesn’t make sense to give these out one by one. The most common permission, I’d say, gives users the ability to Add and Change any entry.

Displaying information in the model admin- In the previous post, anything we added to the “High Schools” model was displayed as “HighSchool object,” which is about the least useful name for anything ever. This is easy to address.

Edit you admin.py file to look like this:

//admin.py
...
class HighSchoolAdmin(admin.ModelAdmin):
    list_display = ('name',)

admin.site.register(HighSchool, HighSchoolAdmin)

All we did there was access the list_display property of our ModelAdmin and tell it to how us the name of the school. Access that model admin section again, and you’ll see “HighSchool object” now reads the actual name of your school. Adding other fields to the page is just a matter of stringing field names together in the same format. For example, an admin displaying all of our fields would just require:

//admin.py
...
    list_display = ('name', 'principal', 'enrollment', 'website')

The order you call them in here denotes the order they appear in the admin.

Making the model admin searchable- Making your model searchable is similarly easy to execute. Again, it requires only one line in the admin:

//admin.py
...
class HighSchoolAdmin(admin.ModelAdmin):
    search_fields = ('name', 'principal')

admin.site.register(HighSchool, HighSchoolAdmin)

Put that in your pipe and smoke it. When you reload the High School page, you’ll see a handy-dandy search box that will search the values of the school name and principal fields.
By default, searches are enclosed in wild cards. So searching for “Smith,” in this example, will return both John Smith High school as well as principal Hymie Goldsmith.

Making fields optional – There is a lot more that can be done with admin.py, but more common problems seem to crop up at the individual record level. Many of these issues need to be addressed in models.py. For instance, let’s say we get three records into our project before we realize that private schools don’t have principals. As things stand, any attempt to leave that field blank will be met with a stern warning from Django saying the Principal field is required.

To make the field optional, we’ll add an argument to its entry in models.py like so:

//models.py
class HighSchool(models.Model):
    name = models.CharField(max_length=50)
    principal = models.CharField(max_length=50, blank=True)
    enrollment = models.IntegerField()
    website = models.URLField(verify_exists=True, max_length=120)

Explicitly allowing blanks means the field is now optional. On the “Add High School” page, you’ll notice the name of the field is in regular typeface as opposed to bold.

Adding some signposts- In my experience, the Django admin is significantly more intuitive than PHPMyAdmin. But that’s still not always enough. Sometimes you need to be able to give your users hints along the way.

This, too, is an easy addition. Let’s say we’ve received a frenzied e-mail from our well-intentioned temp saying he doesn’t know what to put in the “Enrollment” field. He has different counts for each grade, he says, but when he tried to list them, he gets an error saying “Enter a whole number.” Help?

Let’s dive back into models.py and add some help text:

//models.py
class HighSchool(models.Model):
    name = models.CharField(max_length=50)
    principal = models.CharField(max_length=50, blank=True)
    enrollment = models.IntegerField(help_text="Enter one value for the whole school.")
    website = models.URLField(verify_exists=True, max_length=120)

Well hot damn. Out handy-dandy little signpost has been cleanly inserted right under our field, making this thing all but idiot proof. At this point, clicking the “Add High School” button should give you something much like this:


For next time

Between these two posts, you should have enough to rapidly build some simple data entry tools for brand new datasets your organization creates on its own.

Of course, that’s not usually the case. Often, especially in journalism, we need to interface with existing data. Our next installment will cover how to get our arms around legacy datasets and get them into our system.

Switch to our mobile site