Preparing Quran Dataset for RAG

Preparing Quran data is an important first step before building a retrieval-augmented generation (RAG) application. The quality of the retrieval stage depends on the structure, consistency, and reliability of the data prepared beforehand.

This tutorial explores how Quranic text and translations can be inspected, validated, parsed, and combined into a structured dataset suitable for later RAG processing.

This tutorial focuses on data preparation only. Embedding, vector indexing, retrieval, and generation are separate stages of the complete RAG pipeline.

The dataset combines one Arabic Quran text source with two translations. Each source uses the same ayah-level structure, allowing records to be aligned using the surah and ayah numbers. The sources are inspected and validated separately before being combined. Their provenance and licensing terms are documented to maintain traceability to the original sources.

The Arabic Quran text from Tanzil is distributed under the Creative Commons Attribution 3.0 licence. Its terms permit verbatim copying and distribution, but do not permit changing the text. The source must be clearly attributed, a link to Tanzil must be provided, and the required copyright notice must be retained in copies and appropriately reproduced in files derived from or containing a substantial portion of the text.

The English Saheeh International and Indonesian Ministry of Religious Affairs translations available through Tanzil are provided for non-commercial use only. Other uses require permission from the translator or publisher. Tanzil also restricts redistribution of its translation collection on another website unless direct permission is granted.

These terms apply to the third-party source data, not automatically to this tutorial's source code. Refer to the Tanzil Text License and Tanzil Quran Translations for the applicable terms.

The raw Tanzil files use a simple pipe-separated format. Each Quranic verse is represented by its surah number, ayah number, and text.

surah|ayah|text

For example, the three fields represent:

• surah — the surah number.
• ayah — the ayah number within the surah.
• text — the Arabic text or translation.

The files also contain metadata and comment lines beginning with #. These lines are not Quranic verse records and must be skipped during parsing. The following is an illustrative example of the file structure, where actual metadata varies by source.

# metadata
# source information

1|1|...
1|2|...
1|3|...

Validation ensures that the raw data has the expected structure before it is transformed into a dataset. This prevents malformed records from silently entering later stages of the pipeline. The raw records are checked before they are converted into structured data.

• Metadata, comment lines, and empty lines are skipped.
• Each verse record must contain exactly three fields.
• Surah and ayah values are converted to integers.
• Duplicate (surah, ayah) keys are checked during dataset validation.
• Missing values and empty text are checked during dataset validation.

These checks serve different purposes: parsing verifies the input structure, while dataset validation checks the consistency of the resulting records.

Raw file
    │
    ├── Metadata / comments → skip
    ├── Empty lines         → skip
    │
    └── Verse records
            │
            ├── Exactly 3 fields?
            ├── Surah and ayah convertible to integers?
            ├── Duplicate (surah, ayah) keys?
            ├── Missing values?
            └── Empty text?
                    │
                    ▼
              Validated dataset

Parsing converts each valid raw line into structured Python data. The three fields are converted into explicit columns: surah, ayah, and text.

1|1|بِسْمِ اللَّهِ الرَّحْمَٰنِ الرَّحِيمِ
          │
          ▼
{
  "surah": 1,
  "ayah": 1,
  "text": "بِسْمِ اللَّهِ الرَّحْمَٰنِ الرَّحِيمِ"
}

The reusable parsing logic is implemented in read_tanzil_file(). It reads the file, skips non-record lines, validates the record structure, converts numeric fields, and returns a pandas DataFrame.

Each source is first represented as a pandas DataFrame. At this stage, all three sources have the same basic schema.

Because every source uses the same text column name, the column is renamed according to its language before the sources are combined.

arabic = arabic.rename(columns={"text": "arabic"})
english = english.rename(columns={"text": "english"})
indonesian = indonesian.rename(columns={"text": "indonesian"})

The three sources are aligned using the natural Quranic key (surah, ayah). This key identifies a specific ayah independently of the language.

The sources are combined through two sequential inner merges using (surah, ayah) as the join key. This retains only keys present in all three sources, ensuring that each prepared row contains aligned Arabic, English, and Indonesian text.

The prepared dataset contains one row per ayah and keeps the source texts aligned through the (surah, ayah) key.

The merged dataset is validated to check the expected row count, duplicate keys, missing values, and final column structure.

• 6,236 ayah records are present.
• The (surah, ayah) key contains no duplicates.
• No required values are missing.
• The Arabic text and both translations are aligned by the (surah, ayah) key.

The prepared DataFrame is stored as quran.parquet. Parquet is a columnar storage format that preserves the tabular structure while providing efficient storage and loading for analytical workflows.

This Parquet file represents the prepared dataset, not the vector index used for semantic retrieval. Embeddings and vector search are separate stages of the RAG pipeline.

The saved dataset can be loaded back into a pandas DataFrame using load_parquet(). This allows the prepared data to be reused without parsing and combining the original text files again.

loaded_dataset = load_parquet(Path("data/processed/quran.parquet"))
loaded_dataset.head()

The loaded DataFrame retains the prepared dataset structure, including the surah and ayah numbers and the Arabic, English, and Indonesian text columns. To verify that the saved dataset matches the original DataFrame, use equals().

loaded_dataset.equals(prepared_dataset)

A result of True confirms that both DataFrames have the same shape, column labels, data types, and values.

Preparing Quran data for RAG begins with understanding the source files and ensuring that their records can be reliably aligned across languages.

In this tutorial, we have accessed and inspected the Tanzil text files, validated and parsed their records, combined the Arabic text with English and Indonesian translations using the (surah, ayah) key, and saved the resulting dataset as quran.parquet. We also loaded the saved dataset to verify that it could be reused.

The prepared dataset provides a structured foundation for the next stages of ChatQuran, including text processing, embeddings, vector indexing, retrieval, and grounded answer generation.

Related Tutorial: Constructing Documents for RAG Applications

The accompanying video demonstrates the data preparation workflow from the terminal, including inspecting the raw Quran files and validating their structure.