Simulating Searches
This project’s goal was to figure out if someone can perform a good web search. Collaborating with content specialists, psychometricians, and assessment developers, I lead the design and UI/UX to create a simulated search engine.
Focus
After initial pitches and consideration between research, content, and design teams, the focus settled on two specific skills used when performing a web search:
- The initial search engine query.
- Choosing the best site to explore from the standardized search engine results list (comprised of page titles, URLs, and summaries).
We would create a simulated search engine to learn how good people are at these two skills. Our client, the National Center for Educational Statistics (NCES), accepted this design and we moved forward!
Design
I lead the design by proposing a collection of systems to improve the simulative qualities of our search engine. These systems were also chosen based on an earlier market analysis, which found that comparable simulated search engines were particularly weak in fixing user input errors and providing dynamic search results (some engines would spit out the same canned results as long as the users query included one key term).
Weight System
The weighting system was designed to deliver relevant results based on the unique term(s) searched. Content specialists can assign a 0 to 1 weight value for each term and source, ensuring that search results are relevant to the terms searched.
For example, let’s say the weighting was set up as follows
| Term 1* | Term 2 | Term 3 | Term 4 | … | |
| Source 1 | 1 | 0.3 | 0.75 | 0.9 | … |
| Source 2 | 0.5 | 1 | 0.33 | 0.75 | … |
| Source 3 | 0.65 | 0.5 | 1 | 0.33 | … |
| Source 4 | 0.3 | 0.75 | 0.85 | 1 | … |
| … | … | … | … | … | … |
* "Term" can be any text, and multiple forms of a Term can also be inserted. For example, if your Term was "meme", you could enter "meme, memes, dank memes", and the algorithm would know that those 3 comma-separated items would all count as getting that that Term.
I then perform a search for Term 2 and Term 3.
The total weights for those two Terms would be calculated
| Term 2 | Term 3 | Total | |
| Source 1 | 0.30 | 0.75 | 1.05 |
| Source 2 | 1.00 | 0.33 | 1.33 |
| Source 3 | 0.50 | 1.00 | 1.50 |
| Source 4 | 0.75 | 0.85 | 1.60 |
and the user would be displayed the source results in order based on highest-to-lowest total weight values.
While actual search engines like Google are much smarter and multifactorial, it would have been impossible to recreate or use these engines due to their proprietary nature and the constraints of the project (such as needing a walled-garden system that limited the number of sources and Terms so it would be easier to quantify and automate measurement). By implementing this weighting system, we could simulate the core functionality of search engines—matching search queries to sources based on weighted relevance.
In addition to the weighting system, I designed error correction systems that included spell-checking and whitelisting, a suite of front-end parameters to allow for customization, data capture logic, and the overall UI/UX.
Results
The final deliverable, a new form of assessment focused on measuring the first steps of a good web search, was completed on-time and accepted by our client. I wrote an end-to-end design document that covered all aspects of this tool, including
- clear, human-readable descriptions of all underlying systems to be handed off to programming for implementation.
- all aspects of data capture, based on my collaborations with the statisticians and psychometricians who would be using the data.
- user flow and experience.
- parameter requirements to ensure common customizations were easy and didn’t require programming.
- interface styling.
The Simulated Search tool went through user testing, cog labs, a large scale pilot, and is now a part of the National Assessment of Educational Progress. I wrote follow-up documentation covering all aspects of setting up the weighting system and changing various unique parameters, ensuring a clean hand-off.