« Kevin Ngo

Building the Marketplace Feed

20 Sep 2014

The “Feed”, the new feature I spent the last three months grinding out for the Firefox Marketplace. The Feed transforms the FirefoxOS app store to provide an engaging and customized app discovery experience by presenting fresh user-tailored content on every visit. The concept was invented by Liu Liu, a Mozilla design intern I briefly hung out with last year. Well, it quickly gained traction by getting featured on Engadget, presented at the Mozilla Summit, and shown off on prototypes at Mobile World Congress. With more traction came more pressure to ship. We built that ship, and it sailed on time.

Planning Phase

The whole concept had a large scope so we broke it into four versions. For the first version, we focused on getting initial content pushing into the Feed. We planned to build a curation tool for our editorial team to control the content of the Feed, with the ability to tailor different content for different countries. Users would then be able to see fresh and new content on the homepage every day, thus increasing engagement.

Curation Tools

The final product for the Curation Tool, used by the editorial team to feature content and tailor different content for different countries.

Some time was spent on the feature requirements and specifications as well as the visual design. We mainly had three engineers: myself, Chuck, and Davor. It was a slow start. Chuck started early on the technical architecture, building out some APIs and documentation. A few months later, I started work on the curation tool. My initial work was actually done off-the-grid in an RV in the middle of Alaska.

But then the project to optimize the app for the $25 FirefoxOS smartphone took over. Once the air cleared, we all gathered for a work week hosted at Davor’s place in Fort Myers, FL (since I was already in FL for a vacation).

Building Phase

In early June, we had a solid start during the work week with each of the three of us on working on separate components: Chuck on the backend API, Davor on the Feed visuals, me on the curation tools. The face-to-face over remote working was nice. I could ask any question that came to mind about unclear requirements, quickly ask for an API endpoint to be whipped up, or check on how the visuals were looking.

After the foundations were in place, I transitioned to working on all parts of the feature from ElasticSearch, the API, the JS powering the curation tools and the Feed, to the CSS for the newer layout of the site. It was a fun grind for a couple of months just writing and refactoring tons of code, getting dirty with ElasticSearch, optimizing the backend.

The launch date was set for late August. A couple of weeks out, the feature was there, but there were some bugs to iron out and tons of polish to be done. The most sketchy part was having no design mocks for desktop-sized screens so I had to improvise. The last week and a half was a sprint. I split my day up into 6 hours, a break to play some tournament poker at the poker club, and then 2 hours at night. Throw in a late Friday, and we made it to the finish line.

Backend Bits

The Feed needed to have an early focus on scalability. We should be able to add more and more factors (such as where they’re from, what device they’re using, apps they previously installed, content they “loved”) that tailor the Feed towards each user. ElasticSearch lets us easily dump a bunch of stuff into it, and we can do a quick query weighing in all of those factors. We cache the results behind a CDN, throw in couple of layers of client-side caching, and we got a stew going.

Figuring out how to relate data between different indices was a difficult decision. I had several options in managing relations, but chose to manually denormalize the data and manage the relations myself. This allowed for flexibility and fine-tuned optimizations without having to wrestle with ElasticSearch under-the-hood. We had three indices to relate:

This meant three ES queries overall (and 0 database queries). I used the new ElasticSearch Python DSL to help construct queries. Wrote a query (weighing all of the factors about a user) to fetch Feed Items, a query to fetch all of the Feed Elements to attach to the Feed Items, and a query to fetch all of the Apps to attach to the Feed Elements. We throw all of the data to our Django Rest Framework serializers to deserialize the final response.

A complication was having to filter out apps, such as when an app is not public, or when an app is banned from the user’s region, or when an app is not supported on the user’s device. With time constraints, I wrote the filtering code in the view. But after the launch, I was able to consolidate our project’s app filtering code into single query factory, use that, and tweak our serializers to handle filtered apps.

Frontend Bits

With the Feed being made of visual blocks and components, encapsulated template macros kept things clean. These reusable macros could be reused by our curation tools to display previews of the Feed on-the-fly for content curators. We used Isotope.js to arrange the layout of these visual blocks.

Our CSS eventually turned messy, having everything being namespaced by the CSS class .feed, and falling under the trap of OOP-style CSS. A good style guide to follow in the future is @fat’s CSS style guide for Medium.

And since a lot of the code is shared between the frontend of the Firefox Marketplace and the curation tools, I’m currently trying to get our reusable frontend assets managed under Bower.

Conclusion

The Feed is only going to grow in features. It’s a breath of fresh air in comparison to what used to be a never-changing front page on the app store. I’ve seen new apps I’ve never even knew we had and some are actually fun like Astro Alpaca. Hope everyone likes it!