An assistant that answers English questions well and Arabic questions adequately has not been built for a Saudi audience — it has been built for an English one and translated.
Retrieval is where Arabic breaks first
Most retrieval pipelines are tuned on English embeddings. Arabic morphology — prefixes, suffixes, and the same root generating dozens of surface forms — means naïve chunking and matching retrieve the wrong passages confidently. Evaluate retrieval separately from generation, in Arabic, before blaming the model.
Dialect versus Modern Standard Arabic
Users write in dialect and expect answers in MSA. A system trained and evaluated only on MSA input will miss what people actually type.
Diacritics are optional in input, not in meaning
Users almost never type diacritics. Your normalisation has to handle that without collapsing genuinely different words.
Evaluate in Arabic, with Arabic speakers
A bilingual test set written by an English speaker and translated will not surface the failures that matter. Write the Arabic cases natively.
Keep sensitive workloads in the Kingdom
For regulated data, a model deployed in-region or in your own tenancy is often the difference between a pilot and production.