Documents
As the name suggests, documents are a core part of Doctrine — the very reason Doctrine exists is so you can have conversations with your documents. On this page, we'll dive into the different Document endpoints you can use to manage Documents programmatically. We'll look at how to ingest, partition, and query documents.
Properties
- Name
id
- Type
- string
- Description
Unique identifier for the document upload.
- Name
createdAt
- Type
- DateTime
- Description
Timestamp of when the document was uploaded. It is created by default at the time of upload.
- Name
originalUrl
- Type
- string
- Description
The original URL from where the document was uploaded. This is an optional field.
- Name
parsedUrl
- Type
- string
- Description
The parsed URL of the uploaded document. This is an optional field.
- Name
originalText
- Type
- string
- Description
The original text content of the uploaded document. This is an optional field.
- Name
name
- Type
- string
- Description
The name of the uploaded document.
- Name
addedByUserId
- Type
- string
- Description
The unique identifier of the user who uploaded the document.
- Name
organizationId
- Type
- string
- Description
The unique identifier of the organization associated with the uploaded document.
- Name
mimeType
- Type
- string
- Description
The MIME type of the uploaded document. This is an optional field.
- Name
tokens
- Type
- int
- Description
The number of tokens in the uploaded document. This is an optional field.
- Name
partitionId
- Type
- string
- Description
The unique identifier of the partition associated with the uploaded document. This is an optional field.
- Name
metadata
- Type
- Json
- Description
Metadata associated with the uploaded document in the form of a JSON object. This is an optional field.
List all documents
This endpoint allows you to retrieve a paginated list of all your documents. By default, a maximum of ten documents are shown per page.
Optional attributes
- Name
limit
- Type
- integer
- Description
Limit the number of contacts returned.
- Name
skip
- Type
- integer
- Description
Skip x records, useful for pagination.
Request
curl -G https://doctrine.app/api/document \
-H "Authorization: Bearer {token}" \
-d skip=0 \
-d limit=10
Response
[
{
"id": "WAz8eIbvDR60rouK",
"metadata": { ... },
"originalUrl": "https://doctrine.app/Doctrine.pdf",
"parsedUrl": "https://doctrine.app/Doctrine.pdf",
"originalText": "Doctrine is a document management system.",
},
{
"id": "hSIhXBhNe8X1d8Et"
// ...
}
]
Ingest a document
This endpoint allows you to ingest a new document into the system by sending the document's text and name.
Required attributes
- Name
name
- Type
- string
- Description
The name of the document.
Optional attributes
- Name
text
- Type
- string
- Description
The text content of the document.
- Name
url
- Type
- string
- Description
The url of the document to ingest. This can be a link to a PDF, word doc, csv, excel, powerpoint, image etc.,
- Name
websiteUrl
- Type
- string
- Description
The url of the website to ingest. This will ingest the text content of the website.
- Name
mimeType
- Type
- string
- Description
We will figure this out automatically in most scenarios, but you can pass it if you want to.
- Name
partitionId
- Type
- string
- Description
The unique id of the partition you want to ingest the document into.
- Name
partition
- Type
- string
- Description
This is a utility to upsert a partition name if you don't have it's id handly.
- Name
metadata
- Type
- JSON
- Description
Attach metadata to your document.
Request
curl --location --request POST 'http://localhost:3000/api/ingest' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdJZCI6Im9yZ18yT3FEWlFnSHhuY3JCZXdSdEw2M2wzNDlIUEEiLCJ1c2VySWQiOiJ1c2VyXzJESkxnTEgwcEtiM3dhTTFJRGI4dHZ4WmVDaCIsIm5hbWUiOiJUZXN0IiwiaWF0IjoxNjg4MjM5NzYyfQ.lA53AfpzHdGxbTvJvVFzvIMyPXv3p8vQDbX29hbwqRw' \
--header 'Content-Type: application/json' \
--data-raw '{
"text": "Brewing coffee using a Chemex can result in a clean, pure, and flavorful cup of coffee. Here is a simple guide...",
"name": "Sample doc"
}'
Response
{
"id": "WAz8eIbvDR60rouK",
"name": "Sample doc",
"text": "Brewing coffee using a Chemex can result in a clean, pure, and flavorful cup of coffee. Here is a simple guide...",
"createdAt": "2023-07-03T10:20:16.298Z",
"updatedAt": "2023-07-03T10:20:16.298Z"
}
Ask a question
This endpoint allows you to ask a question over the data you've ingested. You can ask the question at different scopes, the default with no documentId or partitionId will ask the question at the organization level scope.
Required attributes
- Name
query
- Type
- string
- Description
The text query to search the documents.
Optional attributes
- Name
documentId
- Type
- string
- Description
Want to talk to a specific document? this is how.
- Name
partitionId
- Type
- string
- Description
Pass this in to ask over a partition, a partition is a collection of documents.
- Name
prompt
- Type
- string
- Description
You can pass in a custom prompt to replace the default prompt. Example prompt:
Given the following sections from our knowledge database, answer the question using only that information, outputted in markdown format. If you are unsure and the answer is not explicitly written in the documentation, say "Sorry, I don't know how to help with that." Context sections: {contextText} Question: """ {query} """
- Name
numberOfDocumentsInContext
- Type
- string
- Description
This controls how many relevant Document snippets are injected into the prompt context, as well as how many are returned with the answer to your question.
Request
curl --location --request POST 'http://localhost:3000/api/query' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data-raw '{
"query": "How long should I let coffee grounds bloom?"
}'
Response
{
"answer": "\nThe coffee grounds should bloom for 30-45 seconds.",
"relevantDocuments": [
{
"id": "cljkeepaj0004332o5iuwr7x5",
"createdAt": "2023-07-01T19:30:34.507Z",
"originalUrl": null,
"parsedUrl": "",
"originalText": "Brewing coffee using a Chemex can result in a clean, pure, and flavorful cup of coffee. Here is a simple guide to brewing coffee using a Chemex:\\n\\nWhat you'll need:\\n\\nChemex\\nChemex filters\\n42 grams of coffee beans\\nGrinder\\n700 grams (or ml) of water, plus additional for rinsing\\nKettle\\nScale\\nTimer\\nSteps:\\n\\nBoil your water: Start by bringing your water to a boil. The ideal brewing temperature is between 195 to 205 degrees Fahrenheit, so let the water cool down a bit after boiling.\\nGrind your coffee: While your water is boiling, grind your coffee beans to a medium-coarse grind. You want the consistency to be similar to sea salt.\\nPrepare the filter: Place the Chemex filter in the brewer with the thicker (multi-layered) side against the pouring spout. Rinse the filter with hot water to get rid of any paper taste and to preheat the Chemex. Then, discard the rinse water.\\nAdd coffee: Place your Chemex on the scale, tare it to zero, and add your ground coffee.\\nBloom the coffee: Start your timer and pour enough water (about 70-100 grams) over the coffee grounds to wet them evenly. Let it sit and \"bloom\" for 30-45 seconds. This allows the coffee to de-gas, enabling the water to yield the full potential of the coffee.\\nContinue pouring: After the bloom, continue pouring the water in a slow, steady spiral from the middle of the grounds outwards, pausing as needed to let it drip through. Try to avoid pouring directly on the filter. Continue until you've reached 700 grams (or ml) of water.\\nWait: Allow the rest of the water to drip through the grounds. This should take around 4 to 5 minutes. If it's dripping too fast, you may need a finer grind. If it's too slow, try a coarser grind.\\nEnjoy: Once the dripping slows to a stop, remove and discard the filter. Swirl the brewed coffee in the Chemex a bit to mix it evenly, then pour and enjoy!\\nRemember, these are just guidelines. The best cup of coffee is one that tastes right to you, so feel free to adjust the amount of coffee, the grind size, the water temperature, and the brew time to suit your tastes. The Chemex is a forgiving brewing method, so don't be afraid to experiment a bit to find your perfect brew.",
"name": "Sample doc",
"addedByUserId": "user_2DJLgLH0pKb3waM1IDb8tvxZeCh",
"organizationId": "org_2OqDZQgHxncrBewRtL63l349HPA",
"mimeType": null,
"tokens": 544,
"status": "processed",
"partitionId": null,
"metadata": null
}
],
"relevantSnippets": [
{
"metadata": {
"id": "cljkeeppv0007332ovn703dn0",
"documentUploadId": "cljkeepaj0004332o5iuwr7x5",
"content": "Brewing coffee using a Chemex can result in a clean, pure, and flavorful cup of coffee. Here is a simple guide to brewing coffee using a Chemex:\\n\\nWhat you'll need:\\n\\nChemex\\nChemex filters\\n42 grams of coffee beans\\nGrinder\\n700 grams (or ml) of water, plus additional for rinsing\\nKettle\\nScale\\nTimer\\nSteps:\\n\\nBoil your water: Start by bringing your water to a boil. The ideal brewing temperature is between 195 to 205 degrees Fahrenheit, so let the water cool down a bit after boiling.\\nGrind your coffee: While your water is boiling, grind your coffee beans to a medium-coarse grind. You want the consistency to be similar to sea salt.\\nPrepare the filter: Place the Chemex filter in the brewer with the thicker (multi-layered) side against the pouring spout. Rinse the filter with hot water to get rid of any paper taste and to preheat the Chemex. Then, discard the rinse water.\\nAdd coffee: Place your Chemex on the scale, tare it to zero, and add your ground coffee.\\nBloom the coffee: Start your timer and pour enough water (about 70-100 grams) over the coffee grounds to wet them evenly. Let it sit and \"bloom\" for 30-45 seconds. This allows the coffee to de-gas, enabling the water to yield the full potential of the coffee.\\nContinue pouring: After the bloom, continue pouring the water in a slow, steady spiral from the middle of the grounds outwards, pausing as needed to let it drip through. Try to avoid pouring directly on the filter. Continue until you've reached 700 grams (or ml) of water.\\nWait: Allow the rest of the water to drip through the grounds. This should take around 4 to 5 minutes. If it's dripping too fast, you may need a finer grind. If it's too slow, try a coarser grind.\\nEnjoy: Once the dripping slows to a stop, remove and discard the filter. Swirl the brewed coffee in the Chemex a bit to mix it evenly, then pour and enjoy!\\nRemember, these are just guidelines. The best cup of coffee is one that tastes right to you, so feel free to adjust the amount of coffee, the grind size, the water temperature, and the brew time to suit your tastes. The Chemex is a forgiving brewing method, so don't be afraid to experiment a bit to find your perfect brew.",
"_distance": 0.1777154337485649
}
}
]
}
Update a document
This endpoint allows you to perform an update on a document.
Optional attributes
- Name
text
- Type
- string
- Description
If you pass in this field, we will delete & update the vectors for this document.
- Name
metadata
- Type
- JSON
- Description
Pass in a metadata JSON object to update the metadata associated with this document.
Request
curl -X PUT https://doctrine.app/api/document/{documentId} \
-H "Authorization: Bearer {token}" \
-d text="Update all my vectors with this text!"
Response
{
"id": "cljkeep9f0002332ox29hf6gp",
"createdAt": "2023-07-01T19:30:34.463Z",
"originalUrl": null,
"parsedUrl": "",
"originalText": "",
"name": "Sample doc",
"addedByUserId": "user_2DJLgLH0pKb3waM1IDb8tvxZeCh",
"organizationId": "org_2OqDZQgHxncrBewRtL63l349HPA",
"mimeType": null,
"tokens": 0,
"status": "processed",
"partitionId": null,
"metadata": null
}
Delete a document
This endpoint allows you to delete documents.
Request
curl -X DELETE https://doctrine.app/api/document/{documentId} \
-H "Authorization: Bearer {token}"