Why Solr?

Solr is a search engine server built with Lucene at its core. Solr’s major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Providing distributed search and index replication, Solr is highly scalable.

New features in Solr 4.0, released last year, include a new web-based user interface a spell checker and support for geolocation data (excellent for news sites wanting to offer geographic search). Given the functionality, choosing Solr was an easy decision, but not one we took lightly. “It is a proven solution, powering huge platforms and excelling,” says Newscoops lead developer Holman Romero. “It is open source and member of the Apache group, with just the set of features you need, and more. From a technical point of view, it is fast and easy to integrate due to its architecture.”

The challenge for Holman and his team (most of whom are based in Prague) was more on how to build the Solr index given the fact Newscoop allows custom article types and fields. To solve this, there is a script that runs periodically (via cron) looking for new content and updating the Solr index accordingly. There is then a listener setup in Newscoop, which allows the same script to run whenever an article is removed.

The second part of the puzzle, was dealing with the search requests coming from website users. Once the user performs a search action via the search form, the request is handled by Newscoop, which prepares the query in a way Solr understands. A request is sent by Newscoop to Solr using its API and Solr sends back the response in JSON format. Newscoop parses the response and sends it to its template engine to finally be rendered to the user.

Setting up Solr

Holman helped me with this small guide on how to install Solr alongside Newscoop. There’s more info on our wiki about requirements and set-up. Feel free to leave questions below and Holman will help as much as he can.

Firstly, you have to download Apache/Solr from Apache and then unpack it in any directory you want to run it from.
$ cp solr-4.1.0.tgz /var/www/
$ cd /var/www
$ tar xvzf solr-4.1.0.tgz

Then we copy over Newscoops solr configuration into Solr itself
$ cp -a /var/www/newscoop/example/solr/* /var/www/solr-4.1.0/example/solr/

It’s important to edit example/solr/solr.xml and add for each language you are using (the name must be the language code) and copy the folder for each of those cores.

To run Solr, head to the example folder and run
$ java -jar start.jar

All you need to do in Newscoop is to enable Solr in the application config file. Open /var/www/newscoop/application/configs/application.ini-dist with
your editor of choice and uncomment this line (by removing the semi-colon at the beginning of the line):
listener[] = search.indexer.article

Newscoop looks for Solr by default in http://localhost:8983/solr. If Solr is running on a different address/port, you can override this by changing the following line in the
/var/www/newscoop/application/configs/application.ini-dist file:
search.solr_server = 'http://<url>:<port>/solr'

(Make sure to add that line to production and CLI environments.)

Now, we need to store Newscoop content in Solr so that search can start. In other words, we need to populate the Solr index with your content. Run the following command manually to get some data for testing.
$ cd /var/www/newscoop
$ php scripts/newscoop.php index:update

In production environments you’d want to set up a cron job to run the same command periodically, so that your Solr index is updated with any new article or article changes in Newscoop.

Newscoop’s templates have a lot more power and customizability than many other CMS templates. If you want to display Solr search results on the front end (to list the most popular hits on the site for example) you have to add a little extra code to your templates. Fields that can be queried include title, type, webcode, authors, topics, keywords, and a custom field name. More on that over on our wiki.

So, there you have it; how to set up a Newscoop site with Solr-powered search. Thanks to Holman for help on the technical guide, Petr for leading the Solr development, and the other Newscoop team members. And, of course, to all the Georgian sites for their testing and feedback during the development process!