feat: add Vectara destination connector

[potter-potter]

Thanks to Ofer at Vectara, we now have a Vectara destination connector.

• There are no dependencies since it is all REST calls to API

[ryannikolaidis]

checking, was there any resolution on validation of the stored and embedded results?

actual should wait for response on "can we test the embedding results"

[potter-potter]

checking, was there any resolution on validation of the stored and embedded results?

@ryannikolaidis I can make a post request with a text query. (but not an embedding) And then have it give the top example. We do get some metadata we can check for. But once the partitioned file hits Vectara, it is a black box. So no guarantee what the are doing as far as embeddings and search.

Example of response:
{ "responseSet": [ { "response": [ { "text": "I confess all these festivities and fireworks are becoming wearisome. \"If they had known that you wished it, the entertainment would have been put off,\" said the prince, who, like a wound-up clock, by force of habit said things he did not even wish to be believed. \"Don't tease! Well, and what has been decided about Novosiltsev's dispatch? <b>You know everything.</b> \"What can one say about it?\" replied the prince in a cold, listless tone. \"What has been decided?", "score": 0.75966287, "metadata": [ { "name": "url", "value": "example-docs/book-war-and-peace-1p.txt" }, { "name": "filename", "value": "book-war-and-peace-1p.txt" }, { "name": "filetype", "value": "text/plain" }, { "name": "last_modified", "value": "2024-01-18T09:39:46" }, { "name": "lang", "value": "eng" }, { "name": "offset", "value": "76" }, { "name": "len", "value": "20" } ], "documentIndex": 0, "corpusKey": { "customerId": 0, "corpusId": 61, "semantics": "DEFAULT", "dim": [], "metadataFilter": "", "lexicalInterpolationConfig": null }, "resultOffset": 344, "resultLength": 20 } ], "status": [], "document": [ { "id": "726c32ce-acf0-471e-88b2-a4a8b040d324", "metadata": [] } ], "summary": [], "futureId": 1 } ], "status": [], "metrics": null }

I looked for some other endpoints that might give us embedding information. But there wasn't anything I found useful.

[ryannikolaidis]

checking, was there any resolution on validation of the stored and embedded results?

@ryannikolaidis I can make a post request with a text query. (but not an embedding) And then have it give the top example. We do get some metadata we can check for. But once the partitioned file hits Vectara, it is a black box. So no guarantee what the are doing as far as embeddings and search.

Example of response: { "responseSet": [ { "response": [ { "text": "I confess all these festivities and fireworks are becoming wearisome. \"If they had known that you wished it, the entertainment would have been put off,\" said the prince, who, like a wound-up clock, by force of habit said things he did not even wish to be believed. \"Don't tease! Well, and what has been decided about Novosiltsev's dispatch? <b>You know everything.</b> \"What can one say about it?\" replied the prince in a cold, listless tone. \"What has been decided?", "score": 0.75966287, "metadata": [ { "name": "url", "value": "example-docs/book-war-and-peace-1p.txt" }, { "name": "filename", "value": "book-war-and-peace-1p.txt" }, { "name": "filetype", "value": "text/plain" }, { "name": "last_modified", "value": "2024-01-18T09:39:46" }, { "name": "lang", "value": "eng" }, { "name": "offset", "value": "76" }, { "name": "len", "value": "20" } ], "documentIndex": 0, "corpusKey": { "customerId": 0, "corpusId": 61, "semantics": "DEFAULT", "dim": [], "metadataFilter": "", "lexicalInterpolationConfig": null }, "resultOffset": 344, "resultLength": 20 } ], "status": [], "document": [ { "id": "726c32ce-acf0-471e-88b2-a4a8b040d324", "metadata": [] } ], "summary": [], "futureId": 1 } ], "status": [], "metrics": null }

I looked for some other endpoints that might give us embedding information. But there wasn't anything I found useful.

Ahh, I see. I think we could but we would need to build and hit the appropriate app in Vectara that also leverages the data we've written.

[ryannikolaidis]

Okay, I think checking count is good enough for now. It would be nice as a future enhancement to actually test end to end with a test application we build in Vectara, but this should at least given us confidence that we're writing to it.