Documents

As the name suggests, documents are a core part of Doctrine — the very reason Doctrine exists is so you can have conversations with your documents. On this page, we'll dive into the different Document endpoints you can use to manage Documents programmatically. We'll look at how to ingest, partition, and query documents.

Properties

  • Name
    id
    Type
    string
    Description

    Unique identifier for the document upload.

  • Name
    createdAt
    Type
    DateTime
    Description

    Timestamp of when the document was uploaded. It is created by default at the time of upload.

  • Name
    originalUrl
    Type
    string
    Description

    The original URL from where the document was uploaded. This is an optional field.

  • Name
    parsedUrl
    Type
    string
    Description

    The parsed URL of the uploaded document. This is an optional field.

  • Name
    originalText
    Type
    string
    Description

    The original text content of the uploaded document. This is an optional field.

  • Name
    name
    Type
    string
    Description

    The name of the uploaded document.

  • Name
    addedByUserId
    Type
    string
    Description

    The unique identifier of the user who uploaded the document.

  • Name
    organizationId
    Type
    string
    Description

    The unique identifier of the organization associated with the uploaded document.

  • Name
    mimeType
    Type
    string
    Description

    The MIME type of the uploaded document. This is an optional field.

  • Name
    tokens
    Type
    int
    Description

    The number of tokens in the uploaded document. This is an optional field.

  • Name
    partitionId
    Type
    string
    Description

    The unique identifier of the partition associated with the uploaded document. This is an optional field.

  • Name
    metadata
    Type
    Json
    Description

    Metadata associated with the uploaded document in the form of a JSON object. This is an optional field.


GET/api/document

List all documents

This endpoint allows you to retrieve a paginated list of all your documents. By default, a maximum of ten documents are shown per page.

Optional attributes

  • Name
    limit
    Type
    integer
    Description

    Limit the number of contacts returned.

  • Name
    skip
    Type
    integer
    Description

    Skip x records, useful for pagination.

Request

GET
/api/document
curl -G https://doctrine.app/api/document \
  -H "Authorization: Bearer {token}" \
  -d skip=0 \
  -d limit=10

Response


  [
      {
          "id": "WAz8eIbvDR60rouK",
          "metadata": { ... },
          "originalUrl": "https://doctrine.app/Doctrine.pdf",
          "parsedUrl": "https://doctrine.app/Doctrine.pdf",
          "originalText": "Doctrine is a document management system.",
      },
      {
          "id": "hSIhXBhNe8X1d8Et"
          // ...
      }
  ]


POST/api/ingest

Ingest a document

This endpoint allows you to ingest a new document into the system by sending the document's text and name.

Required attributes

  • Name
    name
    Type
    string
    Description

    The name of the document.

Optional attributes

  • Name
    text
    Type
    string
    Description

    The text content of the document.

  • Name
    url
    Type
    string
    Description

    The url of the document to ingest. This can be a link to a PDF, word doc, csv, excel, powerpoint, image etc.,

  • Name
    websiteUrl
    Type
    string
    Description

    The url of the website to ingest. This will ingest the text content of the website.

  • Name
    mimeType
    Type
    string
    Description

    We will figure this out automatically in most scenarios, but you can pass it if you want to.

  • Name
    partitionId
    Type
    string
    Description

    The unique id of the partition you want to ingest the document into.

  • Name
    partition
    Type
    string
    Description

    This is a utility to upsert a partition name if you don't have it's id handly.

  • Name
    metadata
    Type
    JSON
    Description

    Attach metadata to your document.

Request

POST
/api/ingest
curl --location --request POST 'http://localhost:3000/api/ingest' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdJZCI6Im9yZ18yT3FEWlFnSHhuY3JCZXdSdEw2M2wzNDlIUEEiLCJ1c2VySWQiOiJ1c2VyXzJESkxnTEgwcEtiM3dhTTFJRGI4dHZ4WmVDaCIsIm5hbWUiOiJUZXN0IiwiaWF0IjoxNjg4MjM5NzYyfQ.lA53AfpzHdGxbTvJvVFzvIMyPXv3p8vQDbX29hbwqRw' \
--header 'Content-Type: application/json' \
--data-raw '{
    "text": "Brewing coffee using a Chemex can result in a clean, pure, and flavorful cup of coffee. Here is a simple guide...",
    "name": "Sample doc"
}'

Response


  {
      "id": "WAz8eIbvDR60rouK",
      "name": "Sample doc",
      "text": "Brewing coffee using a Chemex can result in a clean, pure, and flavorful cup of coffee. Here is a simple guide...",
      "createdAt": "2023-07-03T10:20:16.298Z",
      "updatedAt": "2023-07-03T10:20:16.298Z"
  }


POST/api/query

Ask a question

This endpoint allows you to ask a question over the data you've ingested. You can ask the question at different scopes, the default with no documentId or partitionId will ask the question at the organization level scope.

Required attributes

  • Name
    query
    Type
    string
    Description

    The text query to search the documents.

Optional attributes

  • Name
    documentId
    Type
    string
    Description

    Want to talk to a specific document? this is how.

  • Name
    partitionId
    Type
    string
    Description

    Pass this in to ask over a partition, a partition is a collection of documents.

  • Name
    prompt
    Type
    string
    Description

    You can pass in a custom prompt to replace the default prompt. Example prompt:

    Given the following sections from our knowledge database, answer the question using only that information,
    outputted in markdown format. If you are unsure and the answer
    is not explicitly written in the documentation, say
    "Sorry, I don't know how to help with that."
    Context sections:
    
    {contextText}
    
    Question: """
    {query}
    """
    
  • Name
    numberOfDocumentsInContext
    Type
    string
    Description

    This controls how many relevant Document snippets are injected into the prompt context, as well as how many are returned with the answer to your question.

Request

POST
/api/query
curl --location --request POST 'http://localhost:3000/api/query' \
--header 'Authorization: Bearer ${token}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "How long should I let coffee grounds bloom?"
}'

Response


  {
      "answer": "\nThe coffee grounds should bloom for 30-45 seconds.",
      "relevantDocuments": [
          {
              "id": "cljkeepaj0004332o5iuwr7x5",
              "createdAt": "2023-07-01T19:30:34.507Z",
              "originalUrl": null,
              "parsedUrl": "",
              "originalText": "Brewing coffee using a Chemex can result in a clean, pure, and flavorful cup of coffee. Here is a simple guide to brewing coffee using a Chemex:\\n\\nWhat you'll need:\\n\\nChemex\\nChemex filters\\n42 grams of coffee beans\\nGrinder\\n700 grams (or ml) of water, plus additional for rinsing\\nKettle\\nScale\\nTimer\\nSteps:\\n\\nBoil your water: Start by bringing your water to a boil. The ideal brewing temperature is between 195 to 205 degrees Fahrenheit, so let the water cool down a bit after boiling.\\nGrind your coffee: While your water is boiling, grind your coffee beans to a medium-coarse grind. You want the consistency to be similar to sea salt.\\nPrepare the filter: Place the Chemex filter in the brewer with the thicker (multi-layered) side against the pouring spout. Rinse the filter with hot water to get rid of any paper taste and to preheat the Chemex. Then, discard the rinse water.\\nAdd coffee: Place your Chemex on the scale, tare it to zero, and add your ground coffee.\\nBloom the coffee: Start your timer and pour enough water (about 70-100 grams) over the coffee grounds to wet them evenly. Let it sit and \"bloom\" for 30-45 seconds. This allows the coffee to de-gas, enabling the water to yield the full potential of the coffee.\\nContinue pouring: After the bloom, continue pouring the water in a slow, steady spiral from the middle of the grounds outwards, pausing as needed to let it drip through. Try to avoid pouring directly on the filter. Continue until you've reached 700 grams (or ml) of water.\\nWait: Allow the rest of the water to drip through the grounds. This should take around 4 to 5 minutes. If it's dripping too fast, you may need a finer grind. If it's too slow, try a coarser grind.\\nEnjoy: Once the dripping slows to a stop, remove and discard the filter. Swirl the brewed coffee in the Chemex a bit to mix it evenly, then pour and enjoy!\\nRemember, these are just guidelines. The best cup of coffee is one that tastes right to you, so feel free to adjust the amount of coffee, the grind size, the water temperature, and the brew time to suit your tastes. The Chemex is a forgiving brewing method, so don't be afraid to experiment a bit to find your perfect brew.",
              "name": "Sample doc",
              "addedByUserId": "user_2DJLgLH0pKb3waM1IDb8tvxZeCh",
              "organizationId": "org_2OqDZQgHxncrBewRtL63l349HPA",
              "mimeType": null,
              "tokens": 544,
              "status": "processed",
              "partitionId": null,
              "metadata": null
          }
      ],
      "relevantSnippets": [
          {
              "metadata": {
                  "id": "cljkeeppv0007332ovn703dn0",
                  "documentUploadId": "cljkeepaj0004332o5iuwr7x5",
                  "content": "Brewing coffee using a Chemex can result in a clean, pure, and flavorful cup of coffee. Here is a simple guide to brewing coffee using a Chemex:\\n\\nWhat you'll need:\\n\\nChemex\\nChemex filters\\n42 grams of coffee beans\\nGrinder\\n700 grams (or ml) of water, plus additional for rinsing\\nKettle\\nScale\\nTimer\\nSteps:\\n\\nBoil your water: Start by bringing your water to a boil. The ideal brewing temperature is between 195 to 205 degrees Fahrenheit, so let the water cool down a bit after boiling.\\nGrind your coffee: While your water is boiling, grind your coffee beans to a medium-coarse grind. You want the consistency to be similar to sea salt.\\nPrepare the filter: Place the Chemex filter in the brewer with the thicker (multi-layered) side against the pouring spout. Rinse the filter with hot water to get rid of any paper taste and to preheat the Chemex. Then, discard the rinse water.\\nAdd coffee: Place your Chemex on the scale, tare it to zero, and add your ground coffee.\\nBloom the coffee: Start your timer and pour enough water (about 70-100 grams) over the coffee grounds to wet them evenly. Let it sit and \"bloom\" for 30-45 seconds. This allows the coffee to de-gas, enabling the water to yield the full potential of the coffee.\\nContinue pouring: After the bloom, continue pouring the water in a slow, steady spiral from the middle of the grounds outwards, pausing as needed to let it drip through. Try to avoid pouring directly on the filter. Continue until you've reached 700 grams (or ml) of water.\\nWait: Allow the rest of the water to drip through the grounds. This should take around 4 to 5 minutes. If it's dripping too fast, you may need a finer grind. If it's too slow, try a coarser grind.\\nEnjoy: Once the dripping slows to a stop, remove and discard the filter. Swirl the brewed coffee in the Chemex a bit to mix it evenly, then pour and enjoy!\\nRemember, these are just guidelines. The best cup of coffee is one that tastes right to you, so feel free to adjust the amount of coffee, the grind size, the water temperature, and the brew time to suit your tastes. The Chemex is a forgiving brewing method, so don't be afraid to experiment a bit to find your perfect brew.",
                  "_distance": 0.1777154337485649
              }
          }
      ]
  }


PUT/document/:id

Update a document

This endpoint allows you to perform an update on a document.

Optional attributes

  • Name
    text
    Type
    string
    Description

    If you pass in this field, we will delete & update the vectors for this document.

  • Name
    metadata
    Type
    JSON
    Description

    Pass in a metadata JSON object to update the metadata associated with this document.

Request

PUT
/api/document/{documentId}
curl -X PUT https://doctrine.app/api/document/{documentId} \
  -H "Authorization: Bearer {token}" \
  -d text="Update all my vectors with this text!"

Response

{
    "id": "cljkeep9f0002332ox29hf6gp",
    "createdAt": "2023-07-01T19:30:34.463Z",
    "originalUrl": null,
    "parsedUrl": "",
    "originalText": "",
    "name": "Sample doc",
    "addedByUserId": "user_2DJLgLH0pKb3waM1IDb8tvxZeCh",
    "organizationId": "org_2OqDZQgHxncrBewRtL63l349HPA",
    "mimeType": null,
    "tokens": 0,
    "status": "processed",
    "partitionId": null,
    "metadata": null
}


DELETE/document/:id

Delete a document

This endpoint allows you to delete documents.

Request

DELETE
/document/{documentId}
curl -X DELETE https://doctrine.app/api/document/{documentId} \
  -H "Authorization: Bearer {token}"