Elastic Workplace Search can instantly pull data from a variety of content sources, such as Google Drive, GitHub, and Salesforce. But you may need additional flexibility to meet your own unique organizational data needs.
The Workplace Search custom source API provides a lightweight, API-driven way to extract content into Workplace Search. This flexible tool helps you prevent your custom content from being isolated and show it fully in Workplace Search results as if it came from an off-the-shelf content source integration.
If you’re not familiar with Elastic Workplace Search, please read my previous post:
– Solutions: Elastic Workplace Search: Search all content anytime, anywhere (part 1)
– Solutions: Elastic Workplace Search: Search all content anytime, anywhere (PART 2)
Through these two articles, you will have a clear understanding of Elastic Workplace Search and be able to install Workplace Search properly.
What exactly is a custom source?
Custom sources target enterprise-level sources, in which case the content of the data source is suitable for exposure to many Workplace Search users. Custom sources are typically initially configured by the system administrator, who then provides API credentials for development to push content to Workplace Search. These credentials, along with the authentication provided, identify the source itself to ensure that any pushed data appears in the appropriate custom source.
As a result, custom sources often contain content from the repository, as well as useful information relevant to many individuals in the organization. A simple example might be a CSV file for a contact. A complex example might be a custom repository, such as Trello. We’ll discuss the simple issues here before we tackle Trello’s follow-up issues.
Adding custom sources requires the use of the REST API. This involves some coding, which we chose to implement in Python as needed here. Although no Python skills are required, readers of more technologies are welcome to actually use them.
Start Simple
Enterprise data is usually located in a managed repository, but you may still have old data in file formats such as CSV. This example, while somewhat intentional, allows us to learn about custom resources through a simple tutorial before tackling more complex use cases.
Exposing contact information in Workplace search results can shorten the time it takes a user to find details about colleagues or customers. Here, we assume that these contacts exist as CSV files. Feel free to use our dataset or generate your own. The document used in this example looks like this:
When we create custom sources in Workplace Search, we effectively create a separate container for content. This content will have a unique architecture and its own correlation model. Once configured, the source is exposed to the results just like any other document repository, with all the associated benefits, such as prioritizing dependencies by group.
Create the Custom API source in Workplace Search
If you haven’t already installed your own Workplace Search, see my previous article Solutions: Elastic Workplace Search: Search Everything anytime, Anywhere (I). In that article, there was a detailed introduction to installing Workplace Search.
We open the Workplace Search address and log in:
Click on the Log In:
Click on Launch Workplace Search:
Click Add sources:
Click on Custom API Source:
Enter Customer_Contacts as the name of the Source and click the Create Custom API Source button:
Access tokens and keys are displayed above. The Access Token provides authentication, while the Key uniquely identifies the custom source. Let’s record these two strings separately and use them in the following exercise.
Use the Workplace Search API
Workplace Search provides an intuitive REST API for extracting content to custom sources. In our case, we just need to send the contact to the _bulk_CREATE endpoint. For simplicity, we limit the number of lines in the CSV to 100, because that means we can send all contacts in a single request. If the user has more data or needs more index flexibility, it is recommended that you use one of your favorite languages (such as Python).
Take a quick look at the request interface
Before we start sending the data, let’s take a quick look at the Workplace Search request itself:
POST http://localhost:3002/api/ws/v1/sources/[KEY]/documents/bulk_create
Copy the codeWe need to issue an HTTP POST to the Workplace Search URL. This operation is indicated by the URL suffix document/ Bulk_create. The [KEY] here is the value you indicated above, which uniquely identifies our new source. Documents to be added to this source are sent as JSON in the body payload. In addition, we need to set up two HTTP headers: an authorization header containing the above token (with Bearer prefix) and a Content-Type indicating that the POST body contains JSON. Assuming we use curl, this results in the following request structure:
curl -X POST http://localhost:3002/api/ws/v1/sources/[KEY]/documents/bulk_create \-H "Authorization: Bearer [AUTH_TOKEN]" \-H "Content-Type: application/json" \-d '[
{
"id" : 1,
"first_name" : "Emilio",
"last_name" : "Hughes",
"email" : "[email protected]",
"created_at": "2019-06-01T12:00:00+00:00",
"company": "Magicians Inc",
"job_title": "Procurement Manager"
},
{
"id" : 2,
"first_name" : "Joe",
"last_name" : "Blogs",
"email" : "[email protected]",
"created_at": "2019-06-01T12:00:00+00:00",
"company": "Magicians Inc",
"job_title": "Technical Specialist"
}
]'
Copy the codeRemember to replace your Key and Access Token on it.
After executing the above command:
Click Return to Sources:
Click on the Details:
We can see the two newly imported documents.
At this stage, we do not consider permissions and assume that all users of Workplace Search can find any contact. The above request sends two documents using three different field types: number, date, and text. The number field here is also an ID that uniquely identifies our contact. In practice, this can be more complicated, and appropriate values should be considered.
A little script
As mentioned above, the Workplace Search API requires documentation in JSON format. To convert a CSV file to JSON, we use the popular tool Jq and pass the result to curl:
jq --slurp --raw-input --raw-output \
'split("\n") | .[1:101] | map(split(",")) |
map({"id": .[0],
"first_name": .[1],
"last_name": .[2],
"title": (.[1] + " " + .[2]),
"description": (.[1] + " " + .[2] +" works as a "+.[7] + " at " + .[8] + " in the "+ .[6] +" department. Their email is " + .[3]+"."),
"email": .[3],
"gender": .[4],
"skills": .[5],
"department": .[6],
"job_title": .[7],
"company": .[8],
"url": .[8] })' \
customer_contacts.csv | \
curl -X POST --data-binary @- http://localhost:3002/api/ws/v1/sources/[KEY]/documents/bulk_create \
-H "Authorization: Bearer [API_TOKEN]" \
-H "Content-Type: application/json"
Copy the codeHere, JQ simply splits the file contents with new lines and generates each JSON document using the map function. [1:101] Ensure that we do not consider the first line and do not exceed the 100-document limit specified by the Workplace Search bulk API. Finally, it is passed to curl, which accepts input due to the –data-binary @- argument.
In the command above, we bring in our own Key and Access Token. Execute the command above:
Refresh our Workplace Search. We’ll see something like this. We can see that there are 100 documents.
Custom result display
You may have noticed that in addition to mapping each column in the CSV file to the fields in JSON, we also created the fields title, Description, and URL. While Workplace Search automatically displays this content and uses URL fields to make the results linkable, it also provides an easy way to further optimize the source’s presentation. On the configuration page of the source, navigate to Display Settings. The changes we make here will determine how the contact results are displayed to the user. Given our limited set of fields, the setup required is simple.
Finally, select the Result Detail TAB to modify how the results are displayed when clicked. A little work here can go a long way toward ensuring the best results.
After modification, it looks like this:
Viewing search results
Now we can see our contact data in Workplace Search. These results appear to be fully integrated and benefit from the classic benefits of a unified search experience.
Above we search for Harris Ferrini. If our Workplace includes other sources, such as Google Drive, Salesforce, etc., we might see the following search results:
On the right side of the screen, we can see the Result Detail layout we just customized.
conclusion
By addressing the simple use case of a contact list file on disk first, we showed how easy it is to add enterprise sources to Workplace Search using the “Custom Source” API.