Configuring and Deploying Voyager SolrCloud

The standalone Voyager product is completely self-contained, and each Voyager installation is a separate instance. By contrast, Voyager Cloud takes the same functionality and splits it into two subsystems:

• The Solr index and everything that operates directly on the index
• Everything else, including discovery, processing tasks, the UI and other related components

Getting Started

You’ll need to install and configure the following two components:

• An installation of Apache Zookeeper
• A build of the Voyager Command Line Tools

Downloading and Installing Zookeeper

Zookeeper is the central nervous system of a SolrCloud cluster. Download it from Apache, unless your operating system already has a native version.

• Linux
  Install Zookeeper via the package manager or download the tar package from the Zookeeper site directly and install manually.
  
  
• OSX
  Install Zookeeper via homebrew or download the tar package from the Zookeeper site directly and install manually
  
  
• Windows
  See this documentation from LucidWorks

The Voyager CLI

The Voyager Command Line Interface contains a variety of utilities that allow you to start or stop a Voyager instance and initializing a SolrCloud index. It’s currently the easiest way to set up a Voyager/SolrCloud cluster.

[update CLI location and options, make sure bin directory is in PATH]

Planning the Cluster Topology

When working with SolrCloud, it’s best to plan out the cluster topology ahead of time. Before setup, determine how many index shards you'll require. The SolrCloud documentation contains more detailed info about sharding and other SolrCloud concepts, and we strongly recommend reading through that content before you go any further.

For the purposes of this example, assume we’re going to set up a three node cluster (two Solr nodes and one Voyager/discovery nod)e. There is no limit on the number or Solr  nodes one can create, however currently there can only be a single Voyager/discovery node.

Initializing the Configuration

Each node in the cluster requires its own Voyager data directory. The CLI allows us to easily create an empty data directory:

% voyager init -h

This command takes a few options, the most important of which are:

• The port the node will run on
• The Zookeeper url
• The voyager license

The license parameter can also be set with an environment variable named VOYAGER_LICENSE. To set the VOYAGER_LICENSE variable:

% export VOYAGER_LICENSE=/path/to/my/license.dex

We’ll also assume that Zookeeper is running on localhost on port 2181 (the default on OSX when installed with Homebrew).

With that information, let's create the first Solr node and have it run on port 8001:

% cd /to/some/work/directory
% voyager init -p 8001 -z localhost:2181 solr1

That creates a new data directory named solr1. We’ll create the second directory and set the port to 8002:

% voyager init -p 8002 -z localhost:2181 solr2

Finally we’ll create the data directory for the main voyager/discovery node, running on the standard port 8888:

% voyager init -p 8888 -z localhost:2181 main

Setting Up the SolrCloud Cluster

The next step is to initialize and run the SolrCloud cluster. The first step is to populate/synchronize zookeeper with the configurations of all of the Voyager indexes:

% voyager cloud sync -z localhost:2181

At this point it’s time to start the Solr nodes. Start the first node with the command:

% voyager start -p 8001 -m cloud -s none -d solr1 --stdout

An explanation of the options:

• -p 8001 runs the node on port 8001
• -m cloud starts voyager in “cloud / solr only” mode
• -s none disables security (currently just a workaround, soon to be addressed but for the time being the solr nodes run without any security)
• -d solr1 specifies the data directory to run from
• --stdout enables logs to standard out

Start a new terminal to star the second solr node:

% voyager start -p 8002 -m cloud -s none -d solr2 --stdout

At this point the Solr cluster is running not yet initialized. Visit http://localhost:8001/solr/ in a browser and note that the collection list is empty. A “collection” is one of the Voyager indexes (v0, ssearch, jobs, etc…)

Finally initialize the SolrCloud cluster which will create the collections. This is where the number of shards for each collection is specified. For this example were creating two shards for each collection.

% ./voyager cloud init -s 2 -z localhost:2181 all

The all parameter specifies to initialize all collections. Alternatively we could have run the command with each command name individually.

The init command takes a few seconds to run, while it’s running you should see log activity in the two terminals running the Solr nodes.

At this point the Solr cluster is up and fully functional. Again visit the Solr admin console at http://localhost:8001/solr/. At this point you should see that the collection list has been populated.

Voyager Service on Linux

In some cases, you may want Voyager to run as a background service under Linux. To do this a simple upstart script can be added to /etc/init/voyager-service.conf. The contents of the upstart script are:

description "Voyager service upstart script"
author "Voyager Search"

start on filesystem or runlevel [2345]
stop on shutdown

script
    echo $$ > /var/run/voyager_service.pid
    exec voyager start -p 8001 -m cloud -s none -d solr1
end script

Starting Discovery

The final step in the setup process is to start the voyager/discovery node:

% voyager start -p 8888 -m discovery -d main --stdout

The interesting part of the command is the “-m discovery” switch. It is what causes Voyager to load only the discovery parts and none of the embedded Solr parts.

If all is successful you should see the normal Voyager UI start up.The upstart script can also be used for running discovery as well.

Deploying Discovery on Windows

In some cases, you may want to install Discovery on Windows in order to leverage the Windows-based extractors (Esri, etc.). When installing Discovery on Windows, a cloud.dex file needs to be created to point to the Zookeeper instance and a startup option needs to be defined to put Voyager into discovery mode to load only the discovery parts and none of the embedded Solr parts. To deploy Discovery on Windows, follow these steps:

1. Install Voyager as usual (using the VoyagerServer_x64_with_jre.exe file
2. Create cloud.dex in the {data.dir}/config folder with the following contents (replace {{zookeeper_host}} with the host name of the zookeeper instance):

   {
     "zookeeper":{
       "hosts":[
         "{{zookeeper_host}}:2181"
       ],
       "timeout":5000
     }
   }

3. Open Voyager.vmoptions in the {app.dir} folder and add the following:

-Dmodule.profile=discovery

4. Restart the Voyager Service on Windows