Document Archive

p.1A diagram showing a shoe being captured by a mobile device and an image found display.

Image Found! ADD TO CART $59.98 Run Faster - Running Shoes Also, see Run Longer - Running Shoes!

p.3A diagram showing the 3D Feature Extraction process from different shoe viewpoints.

3D Feature Extractor Combination Component Features Features Features 3D Modeling Component

p.4A diagram of a content provider environment with a user device and third-party provider network.

Content Provider Environment Interface Content Content server Recommend. Engine Image Analysis Component User Data Modeling Component Feature Extractor Training Component

p.5A diagram showing the training process for a neural network with testing modules and query images.

Testing Module Training Module Neural Network Training images Feature Extractor Features Query image

p.5A diagram showing content items being processed through a feature extractor to item features.

Content items Feature Extractor Item Features

p.6A flowchart illustrating the steps of capturing images, processing images, comparing features and presenting search results.

Capture one or more two-dimensional images of a three-dimensional item Process the image(s) to determine three- dimensional features of the image Compare the three-dimensional features to stored three-dimensional features Match Present the search results

p.6A flowchart illustrating the steps of querying a database, comparing distances and associating individual comparisons with a similarity score.

Query a database that includes stored 3D features of catalog items Use a distance technique to compare the query 3D features to stored 3D features Associate individual comparisons with a similarity score Identify a catalog item associated with a highest similarity score

p.7A diagram of a mobile device.

Memory Image Capture Element Processor Communication Component Display Input device

p.8A diagram of a network environment with multiple devices.

Network Web Server Application Server Content Session User Information

p.1This diagram shows an example interface for attribute-based content selection and search, displaying products (shoes) with selectable visual attributes.

Rhinestones Toe Shape Rhinestones Toe Shape Heel Heel Cov Size Strap Type Heel Heel Cov Size Strap Type Select attribute(s) to be modified

p.2This diagram illustrates an attribute-specific disentangled encoder used for content search and selection tasks.

Attribute-specific category color pattern Category vector Conditional similarity retrieval (e.g., search for items with similar color) Complementary content retrieval (e.g., search for complementary item of different category) Query image Attribute manipulation (e.g., change color of item)

p.3This diagram depicts an example architecture for attribute manipulation using an attribute-driven disentangled encoder.

attribute-specific representations manipulation vetcor i Memory Block Target attribute labels vp attribute-specific representations Prototype target embedding compositional triplet loss consistency loss Manipulation: color: navy- beige shared weights Attribute-specific disentangled encoder

p.4This diagram depicts an example architecture for complementary content retrieval.

fc fc ranking loss input image Input category target category concatenation

p.5This is a block diagram of a computing device architecture.

Processing Element Storage Element Transfer App Operating System Sensor Communication I/F Image Sen. SR I/F Wireless Microphone GPS Mobile I/F Wired

p.6This diagram is an example system for sending and providing data.

Data Center Server VM RSVM Manager Server Server VM RSVM Manager Gateway Router Server Manager

p.7This is a flow chart illustrating an example process for attribute-based content selection and search.

Causing to display, on a GUI, an image of a first product, the first product comprising a plurality of visual attributes Receiving a selection of a first visual attribute of the plurality of visual attributes Determining a first plurality of products based at least in part on the first selection of the first visual attribute, wherein the first plurality of products are determined based on a visual similarity to the first product, and based on a visual dissimilarity to the first product with respect to the first visual attribute Causing the first plurality of products to be displayed on the GUI in response to the first selection of the first visual attribute

p.2Example display of content on a computing device where a search query for 'dresses' returns a list of dresses with details like name, brand, and price.

dresses Party Dress by Acme $74.99 Block Dress by ClosetPop $52.50 Short Stripes by HappyClothes $87.99

p.3This figure shows how a keyword-based search for 'jewel neckline dresses' may not return all relevant results due to categorization or classification issues.

Initial image Different version of neckline Similar neckline Not categorized with neckline Categorized as jewel but not jewel

p.3Illustration of different dress necklines.

FIG. 3

p.4A product page displaying supplemental content based on attribute visual similarity.

Butterfly Classic 50's Dress Buy now Price: $59.99 & Free Shipping Size Sweetheart neckline with full circle flared skirt. Fully lined with concealed side zipper. Dresses with similar necklines

p.5Visual attribute similarity pipeline diagram showcasing neckline detection, similarity analysis, and classification for determining high collar.

Neckline Detector Similarity Classifier High Collar

p.5Illustrates the process of finding the neckline region.

FIG. 6

p.5Diagram of attribute space.

FIG. 7

p.6Shows a display to find similar necklines.

dresses Party Dress by Acme $74.99 Block Dress by ClosetPop $52.50 Short Stripes by HappyClothes $87.99 Show similar necklines

p.6An interface to specify attributes.

Show items having similar: Neckline N Y Length N Y Sleeves N Y Shape N Y

p.7A system diagram for attribute-inclusive searches.

Content Provider Environment Third Party Provider Interface Content Query Component Content server Recommend. Engine Visual Attribute Component User Data Weighting Localizer Training Component CNN Component

p.8An example process to train localizer.

Obtain set of catalog images Analyze text associated with selected image Attribute? Add to localizer training set Exclude from localizer training set Full set? Select image from set Analyze image with face detector Face? Use geometric relationship to determine neckline region of image Provide bounding box coordinates and image data to train localizer Done? Process attribute images without detected faces using trained localizer to complete network training set

p.9Shows a process for training neural network.

Obtain network training set of image data from localizer Select training data for image from set Perform feature extraction to generate feature vector Train convolutional neural network for attribute using feature vector Done? Test neural network using remaining portion of training data Provide trained neural network for similarity determinations

p.10A process for finding visual similarity.

Receive query image Determine visual attribute of interest Process query image using trained neural network for attribute Determine location for item, represented in the query image, in attribute space Determine subset of items with representations near item location Rank determined items based on distance to item location in attribute space Provide content for at least a number of highest-ranked items as having similar value for visual attribute of interest

p.11Shows a processing device.

Memory Communication Component Processor Display Input Device

p.4The visual shows an example of how visual search with attribute manipulation works.

Shape: drop Material: metal, gemstone Size: large Color: gold, multi style: Fancy Shape: drop Material: metal, gemstone Size: large Color: blue, green, black style: Fancy

p.5This visual depicts a system environment in which various aspects of the embodiments can be implemented.

Content Provider Environment Network Third Party Provider Interface Content Content server Recommend.Engine Query Component User Data Attribute Manipulation Component Localizer CNN Training Component

p.6This figure shows an example process for visual search with attribute manipulation.

Receive an item selection from a client device Obtain an original feature vector for the selected item Determine attribute to manipulate based on user input Determine attribute to manipulate based on preprogrammed protocol Determine new value for attribute Leave attribute value open Generate a target feature vector with new or open value for the manipulated attribute Query an electronic catalog using the target feature vector Determine a set of candidate items based at least in part on similarity to the target feature vector User data Item data

p.7This image shows a example process visual search with attribute manipulation.

Obtain an original feature vector for a selected item Generate a target feature vector that share only a subset of attribute values with the original feature vector Query an electronic catalog using the target feature vector Determine a set of candidate items based on similarity to the target feature vector

p.8This figure shows an example computing device.

FIG. 8

p.8This diagram is a set of basic components of one or more devices of the present disclosure.

Memory Communication Component Processor Input Device Display FIG. 9

p.9This depicts an example environment for implementing aspects, in accordance with various embodiments.

Network Web Server Application Server Content Session User Information FIG. 10

p.4System Architecture for Search Service Provider.

Search Service Provider Content Search Engine 304 Content Matching and Correlation Engine 302 Association Algorithm 314 Weight Assignment Module 318 Graphing Algorithm 310 Search Results Ranking Engine 308 Association Data 320 Content Search Index 306 Association Graphs 312 Processing Services Web Server 316 Network 324 FIG. 3 .... 300 301

p.5An example association graph showing unique identifiers related to text strings.

a2 a4 402 Unique Identifiers L=M 404 406 406 406 FIG. 4 12 Text Strings 400

p.6Flowchart of a method to return a set of search results, optimized through the application of individually-weighted behavioral signals.

Unique identifier collection module obtains set of one or more unique identifiers for each of multiple items in a dataset ↓ Graph generation module assigns a node to each item, based on the item matching using the unique identifiers ↓ Receive a user search query through an interface ↓ Data mining algorithm extracts one or more keywords from the user search query Determine a set of search results, optimized through applying associations between the dataset items, their unique identifiers, and disaggregated, individually-weighted behavioral signals Using the interface, present the set of search results FIG. 5 500

p.7Block diagram showing the general components of a computing device.

Communication Component 616 Memory 604 Imaging Element 612 Power Component 618 Processor Input Element 602 Audio Element 614 Display Positioning Element 610 608 606 FIG. 6

p.8Block diagram representing a voice communications device architecture.

Automatic Speech Recognition 758 Natural Language Understanding 760 752 752 Processor(s) Processor(s) 754 754 Storage/Memory Storage/Memory 756 756 Communications Circuitry Communications Circuitry 701 Backend 762 764 Applications Text-to-Speech 752 752 Processor(s) Processor(s) 754 754 Storage/Memory Storage/Memory 756 756 Communications Circuitry Communications Circuitry 700 704 Processor(s) 702 Storage/Memory 703 Speech Recognition 705 Communications Circuitry(s) 706 Microphone(s) 708 Speaker(s) 710 Input/Output Interface 712 FIG. 7

p.2A diagram illustrating an example system for brand matching using multi-layer machine learning.

First Itern Data 102- First ML Layer -Second Item Data 104- 106 122 -First Item Vector 114 Second ML Layer -Second Item Vector 116- Source Entity Universal Vector 130 112 124 -Third Item Data 108- -Third Item Vector 118- First ML Layer -Fourth Item Data 110- Second ML Layer -Fourth Item Vector 120- Target Entity Universal Vector 132 First Similarity Vector 144 Second Similarity Vector 146 140 Third ML Layer First Difference Vector 148 142 Fourth ML Layer Second Difference Vector 150 Fifth ML Layer Relationship Score 170 160 FIG. 1 100

p.3A diagram depicting an example system for source brand matching using deep averaging neural networking in multi-layer machine learning.

204 Source Entity Universal Image Vector 206 202 Source Image Vector 1 Image Embedding Source Image Vector 2 DAN Source Image Vector N 208 Source Text Vector 1 210 Text Embedding Source Text Vector 2 DAN Source Text Vector N 220 Source Item 1, Attribute 1 Source Item 2, Attribute 1 Source Item N, Attribute 1 : Source Item 1, Attribute M Source Item 2, Attribute M Source Item N, Attribute M SBERT 230 SBERT Source Item 1, Attribute 1 Vector Source Item 2, Attribute 1 Vector Source Item N, Attribute 1 Vector Source Item 1, Attribute M Vector Source Item 2, Attribute M Vector Source Item N, Attribute M Vector FIG. 2A Source Entity Universal Text Vector 212 222 DAN 232 DAN 200 Source Entity Universal Attribute 1 Vector 224 Source Entity Universal Attribute M Vector 234

p.4A diagram illustrating an example system for target brand matching using deep averaging neural networking in multi-layer machine learning.

252 Target Image Vector 1 254 Image Embedding Target Image Vector 2 DAN Target Entity Universal Image Vector 256 Target Image Vector N 258 Target Text Vector 1 260 Text Embedding Target Text Vector 2 DAN Target Entity Universal Text Vector 262 Target Text Vector N 270 Target Item 1, Attribute 1 Target Item 2, Attribute 1 Target Item N, Attribute 1 : SBERT 280 Target Item 1, Attribute M Target Item 2, Attribute M SBERT Target Item N, Attribute M Target Item 1, Attribute 1 Vector Target Item 2, Attribute 1 Vector Target Item N, Attribute 1 Vector Target Item 1, Attribute M Vector Target Item 2, Attribute M Vector Target Item N, Attribute M Vector FIG. 2B 272 DAN 282 DAN 250 Target Entity Universal Attribute 1 Vector 274 Target Entity Universal Attribute M Vector 284

p.5A diagram illustrating an example system for using twin neural networking in multi-layer machine learning to compare source brand information to target brand information.

302 Source Entity Universal Image Vector 206 Target Entity Universal Image Vector 256 Twin Neural Network 300 Source Entity Universal Image Vector 304 Target Entity Universal Image Vector 306 Source Entity Universal Text Vector 212 Target Entity Universal Text Vector 262 310 Twin Neural Network Source Entity Universal Text Vector 312 Target Entity Universal Text Vector 314 Source Entity Universal Attribute 1 Vector 224 Target Entity Universal Attribute 1 Vector 274 320 Twin Neural Network Source Entity Universal Attribute 1 Vector 322 Target Entity Universal Attribute 1 Vector 324 ... Source Entity Universal Attribute M Vector 234 Target Entity Universal Attribute M Vector 284 330 Twin Neural Network Source Entity Universal Attribute M Vector 332 Target Entity Universal Attribute M Vector 334 FIG. 3A

p.6A diagram illustrating an example system for using difference neural networking in multi-layer machine learning to compare source brand information to target brand information.

352 Source Entity Universal Image Vector 206 Target Entity Universal Image Vector 256 Difference Network 360 Source Entity Universal Text Vector 212 Target Entity Universal Text Vector 262 Difference Network Universal Entity Image Vector Difference 354 Universal Entity Text Vector Difference 362 Source Entity Universal Attribute 1 Vector 224 Target Entity Universal Attribute 1 Vector 274 : Source Entity Universal Attribute M Vector 234 Target Entity Universal Attribute M Vector 284 370 Difference Network 380 Difference Network Universal Entity Attribute 1 Vector Difference 372 Universal Entity Attribute M Vector Difference 382 FIG. 3B 350

p.7A diagram showing an example system for concatenating outputs of the twin neural networking and the difference neural networking and generating a match score.

Source Entity Universal Image Vector 304 Target Entity Universal Image Vector 306 Source Entity Universal Text Vector 312 Target Entity Universal Text Vector 314 Source Entity Universal Attribute 1 Vector 322 Target Entity Universal Attribute 1 Vector 324 : ... 402 400 420 Source Entity Universal Attribute M Vector 332 Target Entity Universal Attribute M Vector 334 Concatenate Concatenated Vector 410 FFN Universal Entity Image Vector Difference 354 Universal Entity Text Vector Difference 362 Universal Entity Attribute 1 Vector Difference 372 Universal Entity Attribute M Vector Difference 382 FIG. 4 Relationship Score 430

p.8A diagram illustrating an example system for brand matching using multi-layer machine learning including client devices, server devices, and networks.

502 NETWORK(S) 580 520 Product Data 530 550 522 ML Layers Image Data 524 Text Data FIG. 5

p.9A flow diagram illustrating a process for brand matching using multi-layer machine learning.

GENERATE, USING UNIVERSAL ENTITY IMAGE EMBEDDINGS AS INPUTS TO A FIRST MACHINE LEARNING (ML) LAYER, IMAGE VECTORS 602 GENERATE, USING UNIVERSAL ENTITY TEXT EMBEDDINGS AS INPUTS TO THE FIRST ML LAYER, TEXT VECTORS GENERATE, USING UNIVERSAL ENTITY PRODUCT DATA AS INPUTS TO THE FIRST ML LAYER, PRODUCT VECTORS GENERATE, USING THE UNIVERSAL ENTITY IMAGE VECTORS AS INPUTS TO A SECOND ML LAYER, A FIRST DIFFERENCE VECTOR GENERATE, USING THE UNIVERSAL ENTITY TEXT VECTORS AS INPUTS TO THE SECOND ML LAYER, A SECOND DIFFERENCE VECTOR GENERATE, USING THE UNIVERSAL ENTITY PRODUCT DATA AS INPUTS TO THE SECOND ML LAYER, A THIRD DIFFERENCE VECTOR GENERATE A CONCATENATED VECTOR BY CONCATENATING THE IMAGE VECTORS WITH THE TEXT VECTORS, THE PRODUCT VECTORS, THE FIRST DIFFERENCE VECTOR, THE SECOND DIFFERENCE VECTOR, AND THE THIRD DIFFERENCE VECTOR GENERATE, USING THE CONCATENATED VECTOR AS AN INPUT TO A THIRD ML LAYER, A RELATIONSHIP SCORE FIG. 6 604 606 608 610 612 614 616

p.10A block diagram of an example machine upon which one or more techniques may be performed.

702 HARDWARE PROCESSOR GRAPHICS 710 DISPLAY DEVICE 724 INSTRUCTIONS 712 INPUT DEVICE MAIN MEMORY 704 724 INSTRUCTIONS UI NAVIGATION 714 DEVICE 706 STATIC MEMORY 724 INSTRUCTIONS STORAGE DEVICE 716 MACHINE- READABLE 722 MEDIUM 724 INSTRUCTIONS 728 SENSORS 720 NETWORK INTERFACE SIGNAL GENERATION DEVICE 718 730 ANTENNA(S) 726 MACHINE LEARNING-530 LAYERS 732 POWER DEVICE 708- COMMUNICATIONS NETWORK FIG. 7 OUTPUT CONTROLLER 734

p.1A diagram illustrating the system architecture for generating quality scores for brand curation.

-100 TARGET BRAND DATA REPOSITORY 114 DATA 120 DATA BRAND PERFORMANCE REPOSITORY 124 DATA 128- 160 130 CONSTRUCTOR UNIT MODEL 134 MODEL REPOSITORY 150 ASSESSOR UNIT SCORES 157 Select 155 SCORE REPOSITORY Brand Index 170 SERVICE DEVICES Non-Select

p.5A diagram illustrating the relevance analyzer routine.

BROWSE RELEVANCE ANALYZER ROUTINE 400 CALCULATE CATEGORY FITNESS 500 SCORES CALCULATE BROWSE SCORES 600 420 410 IS YES CALCULATE THE ITEM NEW NEWNESS SCORE ? 430 NO OBTAIN SALES RANK SCORE 700 CALCULATE SPECIAL PREDICTION BOOST SCORE 440 CALCULATE POPULARITY SCORE 450 CALCULATE PRICE RANGE SCORE Fig.4. CALCULATE BROWSE RELEVANCE SCORE BASED ON CATEGORY FIT SCORE, BROWSE SCORE, POPULARITY SCORE, AND PRICE RANGE SCORE 460 ORGANIZE SCORES INTO A RANKED LIST END 470

p.6Diagram illustrating category fit score subroutine.

500 502 504 CATEGORY FIT SCORE SUBROUTINE DETERMINE ALL POSSIBLE MATCHING CATEGORIES GET FIRST CATEGORY DETERMINE FIT SCORE FOR CATEGORY BASED ON LEVEL AND MATCH NO LAST CATEGORY ? 508 506 YES SORT SCORES IN DESCENDING ORDER BASED ON FIT SCORE 510 RETURN 512 Fig.5.

p.7Diagram illustrating browse score subroutine.

600 602 604 606 608 610 CALCULATE BROWSE SCORE SUBROUTINE GATHER CLICK STREAM DATA FOR ITEM PER BROWSE NODE DETERMINE AMOUNT OF CLICKS ON ITEM DETERMINE NUMBER OF TIMES ADDED TO CART DETERMINE TOTAL TIMES PURCHASED DETERMINE BROWSE SCORE PER BROWSE NODE RETURN 612 Fig.6

p.8Diagram illustrating special prediction boost subroutine.

SPECIAL PREDICTION BOOST SUBROUTINE 702 IS ITEM PART OF A MARKET TREND ? 706 710 NO IS ITEM BASED ON A POPULAR CHARACTER ? NO IS ITEM BASED ON A POPULAR THEME ? NO RETURN 700 704 YES CALCULATE BOOST 708 YES CALCULATE BOOST 712 YES CALCULATE BOOST Fig. 7

p.9A diagram illustrating an example web page.

806 808 810 Browse Dining Drinkware Dinnerware Utensils › All Dining Cookware Cookware Sets Dutch Ovens Saucepans › All Cookware Cutlery Knife Sets 802 Kitchen & Housewares 820 800 822 826 824 Steak Knives Knife Blocks › All Cutlery 810 Housewares 812 Air Conditioners Dining 806 Cutlery 808 Air Purifiers • Coffee Mug Ironing Sewing • Designer Knife Set Small Appliances • Celebrity Bread Machine 806A 810A > All Dining 808A › All Housewares > All Cookware › All Small Applicances Cook's Tools & Gadgets 814 Baking Tools Bar & Wine Tools Spatulas 816 > All Cook's Tools & Gadgets Small Appliances Blenders Bread Machine Coffee Maker Grills Mixers › All Small Appliances 804 Fig.8.

p.3A smartphone displaying a user interface for color selection and product search. The interface shows a captured image of a woman's face, selectable color elements representing different areas of the face, and product listings (lipstick and nail polish) that match the selected color.

108 102 100 140 110 X * 142 116 122 $$$ LIPSTICK 130 RESULTS DONE $$$ 126 Maybelline ★★★★☆ $5.47 Maybelline ★★★★☆ $4.99 L'Oreal Paris ★★★★☆ $6.97 NAIL POLISH 277 RESULTS 106 FIG. 1 112 114 118 120 124 136 130 132 134 138 128 104

p.4A visual representation of different color palettes and their corresponding color names across various brands.

CLAUDIA VERA MICHIYO CHARLOTTE ANGELA ANNVS ANTA JANE CATHERINE NATALE 200 202 204 GRACE OLMA Angel Bombshell Brave Brave Red Brich-a-la GERALDINE LESLE Chatterbus Craving Creme Cup Fabby LANA JEANNE Cirt about Half Hall Hang-up High Strung Hot Gossip ANNABELLA PITA Lady Bug Please Me Plumtul Ravishing Russian Red Alpine Scoo AmazON...AmazOF Speed Up the Amp Sequin Shy Girl Lauder Dial FIG. 2 206

p.5Screenshots of a mobile device's user interface showing the process of selecting a color from an image and initiating a search.

300 320 312 Verison 10:10 AM 312 100% Verison 100 AM 100% Shop by color X Shop by color X 310 328 6 308 334 306 Look around and pick a color or tap to start searching for one virtually in your space. 326 GOT IT 332 330 Color recommending.... 324 302 322 304 FIG. 3A FIG. 3B

p.6Screenshots showing additional steps in selecting a color from an image and displaying search results based on the selected color.

350 340 Vertzon 10x10 AM 100% 352 360 366 Vertzon 10:10 AM 100% 364 354 DONE $$$ 348 LIPSTICK 130 RESULTS Color recommending... 372 346 344 Ravion ★*★*★* $5.47 Maybelitne ★★★★☆ $4.99 NAIL POLISH 277 RESULTS essie ★★★★☆ L'Oreal Paris ★★**** $6.97 O.P.J. assie ★★★★☆ ★★★★☆ 342 362 FIG. 3C FIG. 3D 368 370 374

p.7Screenshots illustrating augmented reality (AR) interfaces for color selection and product search.

400 420 430 eco Verizon 10:10 AM 416 Shop by color 100% X eco Verizon 10:10 AM *100% 414 Shop by color 412 418 410 406 404 428 432 Look around and pick a color or tap to start searching for one virtually in your space. GOT IT 408 L 402 FIG. 4A 426 422 FIG. 4B 424

p.8A screenshot showing a user interface with selectable color elements and search results (e.g., furniture, accessories) displayed in an augmented reality (AR) setting.

448 440 444 BOOCO Verizon 10:10 AM *100% $ $$ HOME 210 RESULTS DONE Casper ★★★★☆ $73 ACCESSORIES Rizzy Home ★★★★★ $40 30 RESULTS Monway ★★★★☆ $781 Racheal Ray Le Creuset HIC Co 442 FIG. 4C 446 450

p.9Screenshots of a user interface showing the process of color picking from a scene.

508 500 510 bec Verizon 10:10 AM *100% X -Color picking... + 506 504 502 FIG. 5A 512

p.10Screenshots illustrating color picking from the AR visual.

528 520 CC Verizon 10:10 AM *100% X Color picking... 526 524 530 ও 522 FIG. 5B

p.11A graphical representation of a color database map.

200 100 0 -100 -200 -300 -500 -400 -300 -200 -100 0 100 FIG. 6

p.12A graphical representation of an example product color database map.

40 20 0000 0 -20 -40 -40 -20 100000 0 20 40 FIG. 7

p.This diagram illustrates a process for receiving and processing natural language input to perform an action.

Receive first data representing a natural language input 130 Using a first trained classifier, determine the natural language input is a complex natural language input 132 Semantically tag the complex natural language input 134 Based on the semantic tags, identify, in the complex natural language input, a first query and a second query dependent on the first query 136 Determine the first query is to undergo information retrieval processing 138 Perform information retrieval processing on the first query to determine a first query result 140 Populate the second query with the first query result 142 Determine the populated second query is to be processed using intent classification processing 144 Perform intent classification processing on the populated second query to perform an action responsive to the complex natural language input 146

p.1.0000000000000002

p.5Diagram showing an example process for mapping review articles to a marketplace catalog, from obtaining reviews and images to training an assignment network.

Obtain a training text and a training image, the training text and the training image being associat- ed with a location in an item repository Generate, from the training text using a text autoencoder, a training text short code Generate, from the training image using an image autoencoder, a training image short code Generate a training short code from the training text short code and the training image short code Train an assignment network using at least the training short code as an input and the location in the item repository as a target output, yielding a trained assignment network

p.3Diagram illustrating training autoencoders using product reviews and images to generate text and image short codes.

Stacked LSTM Autoencoder Word ► Sentence Paragraph Review Resize Images Convolutional Autoencoder

p.4Diagram showing the training of an assignment network using the generated training short code to determine relevant attributes

Attributes

p.7A computing device with the components for the methods and systems presented.

Memory Processor Network Interface Component Display Input Device

p.1The image displays a tablet interface showing search results for "Chairs" with filters and different chair types displayed as products.

Chairs, Filters: Price, Material, Color, Brand A Love Seat, Pull Out Sofa, End Table, Lawn Chair, Recliner, Throw Pillow, Desk Chair, Folding Chair

p.1The image displays a tablet interface showing search results for "Office Chairs" with filters and different office chair types displayed as products.

Office Chairs, Filters: Price, Material, Color, Desk Chair, Executive Desk Chair, Ergo Chair, Desk Chair, Basic Chair, Office Chair, Brand A Love Seat, Desk Chair

p.3This diagram illustrates a computing device providing access to an electronic marketplace with different categories of chairs.

Chairs, Best-sellers, Chairs for Office, Chairs for living room, Chairs with leather, Camping chairs, See more

p.4This illustrates a high-level diagram of the search and content delivery environment.

Network, Third Party Provider, Resource Provider Environment, Interface, Query Receiver, Content Engine, Content, Display Content Generator, Virtual Shelf Engine, Layout Engine

p.5The diagram illustrates the components of the virtual shelf engine.

Virtual Shelf Engine, Queries, Search History, Reform History, Q-S Tables, Shelf Selector, Sponsored Products, Machine Learning System, ASIN, Diversity Module, Intent Module, Ranking Selector, Rankers, Ranking Generator, Shelf Ranker

p.6The image shows an example data table for implementing aspects of the embodiments of the disclosure.

Query, Shelf type 1, Shelf type 2, Shelf type N, Shelf X, Shelf 1a, Shelf 2a, Shelf Na, Shelf Y, Shelf 1b, Shelf 2b, Shelf Nb, Shelf Z, Shelf 1n, Shelf 2n, Shelf Nn, Shelf N

p.6This diagram visualizes the structure for query data and corresponding shelves for chairs.

"Chairs", Rooms, Material, Type, Material, Living room, Wood, Armchair, N Room, Dining room, Leather, Folding, Brand A, N room, N, N, N

p.7This diagram depicts a standard query and the returned results, followed by a more detailed breakdown of a query using virtual shelves to rank and refine results.

QUERY, ASIN, X RESULTS, QUERY, VS, RANKERS, VS₁ = Y RESULTS, VS2 = Z RESULTS, VSN = N RESULTS, VS₁ = Y-A RESULTS, VS2 = Z-A RESULTS, VSN = N-A RESULTS

p.8The diagram provides an overview of how virtual shelves interact with rankers, in particular feature comparisons, in order to rank relevant items.

VS₁, VS2, VSN, ASIN-1, ASIN-1, ASIN-2, FEAT-1, FEAT-2, FEAT-4, FEAT-2, FEAT-4, FEAT-3, FEAT-3, FEAT-5, FEAT-5, ASIN2, ASIN3, ASIN4, FEAT-1, FEAT-2, FEAT-3, FEAT-2, FEAT-4, FEAT-4, FEAT-3, FEAT-5, FEAT-5, R1, R2, RN, FEAT-1, FEAT-2, FEAT-3, FEAT-2, FEAT-4, FEAT-4, FEAT-3, FEAT-5, FEAT-5, FEAT-4, FEAT-6, FEAT-7

p.9The flowchart depicts the steps involved in providing one or more virtual shelves in response to a broad query.

Receive search query, Determine search query is a broad query, Generate one or more virtual shelves, Select one or more products for inclusion on the one or more virtual shelves, Rank the one or more virtual shelves, Present the one or more virtual shelves

p.10This flowchart outlines a decision making process with search queries to determine historical search reformulations.

Receive search query, Broad query?, No, Return search results based one on or more standard criteria, Yes, Receive historical search data, Determine historical search reformulations, Reformulations above threshold?, Yes, Determine relationships between historical search reformulations, Generate one or more virtual shelves

p.11The flowchart depicts the steps of checking a broad query, obtaining historical data and ASIN metadata to generate virtual shelves.

Receive search query, Broad query?, No, Return search results based one on or more standard criteria, Yes, Obtain historical data and ASIN metadata, Determine clustering data for ASIN, based at least in part on the historical data and the metadata, Generate one or more virtual shelves

p.12This flowchart illustrates an example process for populating one or more virtual shelves.

Receive virtual shelf information, Obtain ranker properties, Determine searcher intent, Select ranker, based at least in part on the searcher intent, Apply ranker to products associated with the virtual shelf, Determine a product ordering for the virtual shelf

p.13Illustrates front and back views of an example computing device with capture elements.

Front, Back

p.13This is a component view of the computing device used to apply the current patent.

Memory, Image Capture Element, Processor, Networking Components, Display, Input Device

p.14This is an architectural overview of the search and content delivery system.

Network, Web Server, Application Server, Content, Session, User Information

p.3A schematic representation of a display page presented by a web server, showing a list of items with associated graphics, descriptions, and interactive options.

File Edit View Go Favorite Help A Back Forw... Stop Refresh Home Search Favorite Print Font Mail Address 10 15- Gifts for Graduates 20 25 45 30 GRAPHIC ITEM 1 VIEW SIMILAR 1 ITEMS 20 45 25 30 GRAPHIC ITEM 2 VIEW SIMILAR 2 ITEMS 20 25 45 30 GRAPHIC ITEM 3 VIEW SIMILAR 3 ITEMS 20 25 45 30 GRAPHIC ITEM 4 VIEW SIMILAR 4 ITEMS 35 MORE FIG. 1A

p.4A system diagram outlining the components and data storage used for context-based ranking.

110 120 8888 USER WEB BROWSER SERVER 140 QUERY LOG 130 ANALYSIS DISPLAY TRACKING COMPONENT SERVICE ACTIVITY CONTEXT EXPOSURE DATABASE DATABASE 155 150 135 160 165 WEIGHTING WEIGHTING SERVICE DATABASE 180 WEIGHT LIST ORDERING TABLE MODULE FIG. 1B

p.5A data structure for the activity database, displaying items and user activities associated with each context.

200a 200b 200n CONTEXT 1 CONTEXT 2 CONTEXT N 2100 ITEM 1 ITEM 1 ITEM 1 DETAIL PAGE -220a DETAIL PAGE DETAIL PAGE VIEWS VIEWS VIEWS CART ADDS 220b CART ADDS CART ADDS WISHLIST ADDS 220n WISHLIST ADDS WISHLIST ADDS 210b ITEM 2 ITEM 2 ITEM 2 DETAIL PAGE DETAIL PAGE DETAIL PAGE VIEWS VIEWS VIEWS CART ADDS CART ADDS CART ADDS WISHLIST ADDS WISHLIST ADDS WISHLIST ADDS ITEM X ITEM X DETAIL PAGE DETAIL PAGE ITEM X VIEWS VIEWS DETAIL PAGE VIEWS CART ADDS CART ADDS CART ADDS WISHLIST ADDS 210n WISHLIST ADDS WISHLIST ADDS FIG. 2

p.6A flowchart demonstrating the process of generating weights for individual items by the weighting service.

300 START SELECT NEXT ITEM 310 READ EXPOSURE DATA FROM 320 EXPOSURE DATABASE READ ACTIVITY DATA FROM ACTIVITY DATABASE -325 CALCULATE ACTIVITY 330 INDEX READ HISTORICAL DATA FROM 340 WEIGHTING SERVICE DATABASE CALCULATE -345 RAW WEIGHT CALCULATE INTERMEDIATE WEIGHT 350 -380 YES 370 NO ARE 360 THERE MORE ITEMS ? CALCULATE NEW HISTORICAL 355 EXPOSURE AND ACTIVITY STORE HISTORICAL EXPOSURE AND ACTIVITY IN WEIGHTING DATABASE REDISTRIBUTE INTERMEDIATE WEIGHTS TO PRODUCE FINAL WEIGHTS STORE FINAL WEIGHTS 385 390 END IN WEIGHT TABLE FIG. 3

p.7A flowchart demonstrating the process of the web server requesting the ordering of a list of items from the list ordering module.

120 405 410 180 WEB SERVER LIST ORDERING MODULE DETERMINE CONTEXT FOR LIST 420 DETERMINE ITEMS IN LIST SEND LIST AND CONTEXT TO LIST ORDERING ELEMENT 455 RECEIVE ORDER OF ITEMS FROM LIST ORDERING ELEMENT 460 USE ORDER TO GENERATE DISPLAY 425 430 RECEIVE LIST AND CONTEXT RETRIEVE WEIGHTS FOR ITEM/ CONTEXTS FROM WEIGHT TABLE 440 ORDER ITEMS BY WEIGHT 445 PERFORM ADDITIONAL ORDERING (OPTIONAL) 450 RETURN ORDER TO WEB SERVER FIG. 4

p.3A pictorial diagram illustrating an example of a physical space with additional items, and a search bar, on a computing device.

HTTPS://WWW.FIT-MY-STUFF.COM 1 125 shoe rack 120 100 110

p.4A pictorial diagram illustrating an imaginary bounding box that limits the size of additional items to be placed in the physical space.

HTTPS://WWW.FIT-MY-STUFF.COM 1 + 125 shoe rack FIG. 1B 130 120 100 110

p.5A pictorial diagram of a sample preview image illustrates how the selected additional item may be placed in the physical space.

HTTPS://WWW.FIT-MY-STUFF.COM 125 shoe rack 100 + 140 ITEM 1 IMAGE ITEM 2 IMAGE 150 ITEM 4 IMAGE 160 FIG. 1C ITEM 5 IMAGE 120 ITEM 6 IMAGE 110

p.6A block diagram depicts an illustrative operating environment for identifying items that satisfy certain dimensional criteria.

USER DEVICES 202 200 220 NETWORK 204 SERVERS INTERACTIVE COMPUTING SYSTEM 210 230 SEARCH SERVICE 270 240 ELECTRONIC CATALOG SERVICE ITEM DATA REPOSITORY -272 ITEM ATTRIBUTES 260 RECOMMENDATION SERVICE 250 ITEM IDENTIFICATION SERVICE DIMENSION INFORMATION EXTRACTOR ITEM LOCATOR FIG. 2A 252 254 274 USER DATA REPOSITORY

p.7A general architecture of a computing device providing an item identification service for determining dimensional criteria and identifying a set of items that satisfy the determined dimensional criteria.

ITEM IDENTIFICATION SERVICE 290 MEMORY PROCESSING UNIT NETWORK INTERFACE COMPUTER READABLE MEDIUM DRIVE INPUT/OUTPUT DEVICE INTERFACE 250 280 282 USER INTERFACE UNIT 292 284 OPERATING SYSTEM 294 252 DIMENSION INFORMATION EXTRACTOR 254 296 FIG. 2B ITEM LOCATOR

p.8A flow diagram of an illustrative method implemented at least in part by an item identification service for determining dimensional criteria and outputting a set of items that satisfy the determined dimensional criteria.

START ACCESS SEARCH CRITERIA ASSOCIATED WITH A SEARCH REQUEST 302 304 ANALYZE THE SEARCH CRITERIA TO DETERMINE DIMENSIONAL CRITERIA DEFINING A PHYSICAL SPACE DETERMINE A SET OF ITEMS THAT SATISFY THE DIMENSIONAL CRITERIA OF THE PHYSICAL SPACE OUTPUT THE SET OF ITEMS FOR PRESENTATION TO A USER END FIG. 3A

p.9A flow diagram of an illustrative method implemented at least in part by an item identification service for identifying items that satisfy certain dimensional criteria.

START GENERATE A COMPOSITE KEY BASED ON THE DIMENSIONAL CRITERIA OF THE PHYSICAL SPACE, WHERE THE COMPOSITE KEY INCLUDES ONE OR MORE OF THE LENGTH, WIDTH, HEIGHT OR VOLUME OF THE PHYSICAL SPACE 306A SEARCH, USING THE COMPOSITE KEY, FOR CANDIDATE ITEMS IN ONE OR MORE DATABASES THAT EACH STORE A PLURALITY OF SORTED ITEM KEYS 306B COMBINE SEARCH RESULTS OBTAINED FOR THE COMPOSITE KEY AND REMOVE ANY DUPLICATE ITEMS END FIG. 3B

p.1This diagram shows an example of an web browser and the search result for Brand A shoes.

WEB BROWSER EILE EDIT VIEW FAVORITES TOOLS HELP ADDRESS WWW.SEAROLEXAMPLE.COM GO SEARCH.EXAMPLE.COM BRAND A SHDES DEPARTMENT CLOTHING, SHOES AND JEWELRY "BRAND A SHOES" BOY GIRL SIMILAR SEARCHES: BRAND A SNEAKERS. BRAND A SHOWING FIRST 10 OF 115 RESULTS WOMEN MEN UNIFORMS, WORK AND SAFETY BABY 1. BRAND A-SNEAKERS $52,99 COSTUMES AND ACCESSORIES +SEE ALL 28 DEPARTMENTS QUICKSHIP AVAILABLE ★★★★☆ (123 REVIEWS) BRAND ANY BRAND SIZE 7 SIZE 8 SIZE 9 SIZE 10 SIZE 11 BRAND A (13) RED $15.99 $15.99 N/A $21.99 N/A BRAND B (23) GREEN $18.99 $15.99 $16.99 $21.99 N/A BRAND C (2) BLUE $15.99 $15.99 N/A $21.99 $22.99 BRAND D (31) 2. BRAND A-BABY SHOES +MORE BRANDS $19.99 SIZE 7(156) QUICKSHIP AVAILABLE ***** (141 REVIEWS) 8 (145) SEE VARIATIONS: COLOR, OFFERED BY ELECTRONIC 9(44) MARKETPLACE 10 (18) SIZE X COLORX 11 (16) SIZE 7 SIZE 8 SIZE 9 SIZE 10 SIZE 11 +MORE SIZES GRAY N/A $22.99 N/A $21.99 N/A BLACK 22.99 N/A N/A N/A N/A COLOR WHITE NA N/A N/A N/A N/A (BLUE) (RED) 3. BRAND D-BOWLING SHOES $29.99 (BLACK) (WHITE) QUICKSHIP AVAILABLE ★★★★☆ (3141 REVIEWS) PRICE 4. BRAND C-HIGH TOP UNDER $25 (23) $44.99 $25-$50 (37) (SOLD BY THIRD PARTY CO.) $50-$100 (41)

p.2This diagram shows an example of an web browser and the search result for liquid laundry soap.

WEB BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP ADDRESS WWW.SEARCH.EXAMPLE.COM GO "LIQUID LAUNDRY SOAP" SEARCH.EXAMPLE.COM SIMILAR SEARCHES: BRAND A SINGLE PACKS, POWDER LAUNDRY SOAP BRAND A BRAND A LAUNDRY DETERGENT QUICKSHIP AVAILABLE (3141 REVIEWS) MORE VARIATIONS: PRESENTATION TYPE, MERCHANT 12 oz 64 oz 72 oz 84 oz NOT AVAILABLE NOT AVAILABLE NOT AVAILABLE $16.99 $21.99 $22.99 MOUNTAIN SPRING NOT AVAILABLE POWDER FRESH FLORAL NOT AVAILABLE NOT AVAILABLE NOT AVAILABLE NOT AVAILABLE BRAND C LAUNDRY DETERGENT-32 oz. $22.95 (253 REVIEWS)

p.3The diagram shows the environment and the architecture of a dynamic query variations engine

PROCESSOR(S) 214 MEMORY 212 APPLICATION 206 -204(1) -204(N) ... USER DEVICE(S) USER(S) 202 NETWORK(S) 208 SERVICE PROVIDER COMPUTERS 210 O/S 226 MEMORY 216 DYNAMIC QUERY VARIATIONS ENGINE 102 DATA STORE 228 PROCESSOR(S) 218 STORAGE 220 COMM. CONN. 222 I/O DEVICE(S) 224

p.4The diagram illustrates the connections of different engine modules and stores related to the dynamic query variation engine.

DYNAMIC QUERY VARIATIONS ENGINE 102 USER PROFILE DATA STORE 306 APPLICATION PROGRAMMING INTERFACE GRAPHICAL USER INTERFACE 312 314 QUERY ANALYSIS ENGINE 320 HISTORICAL ANALYSIS ENGINE 316 DISPLAY MANAGER 322 INVENTORY DATA STORE 308 SEARCH QUERY DATA STORE 310

p.5The diagram shows an example of how the data for the inventory and the brands for the products are organized.

INVENTORY BRAND A SHOES RED, SIZE 8, SNEAKER BLUE, SIZE 8, SNEAKER GREEN, SIZE 7, HIGH TOP BRAND B SHOES BLACK, SIZE 11, OXFORD BLACK, SIZE 10, LOAFER BROWN, SIZE 12, TUXEDO SHOE BRAND C SHOES WHITE, SIZE 7, PUMP RED, SIZE 9, HIGH HEEL BLACK, SIZE 9, PLATFORM SHOE

p.6This diagram shows an example of an interface with the shoes search results.

WEB BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP ADDRESS WWW.SEARCH.EXAMPLE.COM SEARCH.EXAMPLE.COM SHOES DEPARTMENT CLOTHING, SHOES AND JEWELRY "SHOES" GO SIMILAR SEARCHES: BRAND A SHOES, SHOES BRAND A SHOWING FIRST 10 OF 253 RESULTS ΒΟΥ GIRL WOMEN MEN BRAND D-BOWLING SHOES UNIFORMS, WORK AND SAFETY $29.99 QUICKSHIP AVAILABLE BABY (3141 REVIEWS) COSTUMES AND ACCESSORIES +SEE ALL 28 DEPARTMENTS BRAND E BABY OXFORDS $19.99 BRAND ANY BRAND QUICKSHIP AVAILABLE BRAND A (13) (141 REVIEWS) BRAND B (23) BRAND A-SNEAKER BRAND C (2) $52.99 BRAND D (31) QUICKSHIP AVAILABLE +MORE BRANDS SIZE 7 (156) (123 REVIEWS) 8 (145) BRAND C-HIGH TOP 9 (44) $44.99 10 (18) (SOLD BY THIRD PARTY CO.) 11 (16) (31 REVIEWS) +MORE SIZES BRAND B - MEN'S OXFORD COLOR $19.99 QUICKSHIP AVAILABLE (437 REVIEWS) BRAND F PLATFORM PUMPS $89.99 PRICE UNDER $25 (23) $25-$50 (37) (SOLD BY THIRD PARTY CO.) $50-$100 (41) (1234 REVIEWS)

p.7This diagram shows an example of an interface with the Brand A shoes search results.

WEB BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP ADDRESS WWW.SEARCH.EXAMPLE.COM SEARCH.EXAMPLE.COM BRAND A SHOES DEPARTMENT CLOTHING, SHOES AND JEWELRY "BRAND A SHOES" SIMILAR SEARCHES: BRAND A SNEAKERS, HIGH TOPS BRAND A SHOWING FIRST 10 OF 253 RESULTS ΒΟΥ GIRL WOMEN MEN BRAND A-SNEAKERS UNIFORMS, WORK AND SAFETY $52.99 BABY QUICKSHIP AVAILABLE COSTUMES AND ACCESSORIES +SEE ALL 28 DEPARTMENTS (123 REVIEWS) RED SIZE 7 $15.99 SIZE 8 $15.99 SIZE 9 N/A SIZE 10 $21.99 SIZE 11 N/A GREEN $18.99 $15.99 $16.99 $21.99 N/A BLUE $15.99 $15.99 N/A $21.99 $22.99 BRAND ANY BRAND BRAND A (13) BRAND B (23) BRAND C (2) BRAND A BABY SHOES BRAND D (31) $19.99 +MORE BRANDS QUICK SHIP AVAILABLE SIZE 7 (156) (141 REVIEWS) 8 (145) SEE VARITIONS: SIZE, COLOR, OFFERED BY THE ELECTRONIC MARKETPLACE 9 (44) 10 (18) SIZE X COLORX 11 (16) SIZE 7 +MORE SIZES SIZE 8 N/A SIZE 9 SIZE 10 SIZE 11 $22.99 N/A $21.99 N/A GRAY BLACK 22.99 N/A N/A N/A N/A COLOR WHITE N/A N/A N/A N/A N/A (BLUE) (RED) BRAND D-BOWLING SHOES $29.99 (BLACK) (WHITE) QUICKSHIP AVAILABLE (3141 REVIEWS) PRICE UNDER $25 (23) BRAND C-HIGH TOP $44.99 $25-$50 (37) (SOLD BY THIRD PARTY CO.) $50-$100 (41) (31 REVIEWS)

p.9This diagram shows an example of an interface with the Brand A, Size 8 search results.

WEB BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP ADDRESS WWW.SEARCH.EXAMPLE.COM BRAND A, SIZE 8 SEARCH.EXAMPLE.COM DEPARTMENT CLOTHING, SHOES AND JEWELRY "BRAND A, SIZE 8" GO SIMILAR SEARCHES: BRAND A SIZE 8 SNEAKERS, SHOES BRAND SHOWING FIRST 10 OF 56 RESULTS ΒΟΥ GIRL WOMEN MEN BRAND A-SNEAKERS UNIFORMS, WORK AND SAFETY $52.99 BABY COSTUMES AND ACCESSORIES QUICKSHIP AVAILABLE +SEE ALL 28 DEPARTMENTS BRAND ANY BRAND (123 REVIEWS) BRAND A (13) RED SIZE 7 $15.99 SIZE 8 $15.99 SIZE 9 SIZE 10 SIZE 11 N/A $21.99 N/A BRAND B (23) GREEN $18.99 $15.99 $16.99 $21.99 N/A BRAND C (2) BLUE $15.99 $15.99 N/A $21.99 $22.99 BRAND D (31) BRAND A BABY SHOES +MORE BRANDS $19.99 SIZE 7 (156) QUICKSHIP AVAILABLE 8 (145) (141 REVIEWS) 9 (44) SEE VARIATIONS: OFFERED BY THE ELECTRONIC MARKETPLACE PROVIDER 10 (18) SIZE X COLOR X OFFERED BY X 11 (16) +MORE SIZES OFFERED BY SIZE 8 COLOR GRAY $22.99 BLACK N/A (BLUE) (RED) WHITE N/A (BLACK) (WHITE) BRAND D-BOWLING SHOES PRICE $29.99 UNDER $25 (23) QUICKSHIP AVAILABLE $25-$50 (37) (3141 REVIEWS) $50-$100 (41) BRAND C-HIGH TOP $44.99 (SOLD BY THIRD PARTY CO.) (31 REVIEWS)

p.2Diagram of the operating environment including data stores, service server, and a client device.

DATA STORE(S) 102 DATA STORE-INTERFACE COMPONENT 112 104 SERVICE (SERVER) 110- 106- NETWORK CLIENT DEVICE USER-INTERFACE COMPONENT Fig.1. 108 100

p.3Diagram illustrating interactions when a client device submits a search query and the server returns search categories ordered by rank information.

1002 (4) RELEVANCE SEARCH RESULT 102 DATA STORE(S) 104 SERVICE (SERVER) (3) REQUEST FOR FIRST SEARCH NETWORK (2) REQUEST FOR FIRST SEARCH CLIENT DEVICE (1) USER SELECTION OF FIRST SEARCH Fig.2A. 106 108 (5) ORDERED SET OF VALUES FOR "SEARCH CATEGORIES" BASED ON RELEVANCE SEARCH RESULT

p.4Diagram illustrating further interactions of components for a second search based on search categories.

100 (11) SEARCH RESULT 102 DATA STORE(S) 104 SERVICE (SERVER) (12) NEW ORDERED SETS OF VALUES FOR "SEARCH CATEGORIES" BASED ON SEARCH RESULT (10) REQUEST FOR SECOND SEARCH! NETWORK (6) SEARCH RESULTS, "SEARCH CATEGORIES" &ORDERED SETS OF VALUES 106 (9) REQUEST FOR SECOND SEARCH CLIENT DEVICE (7) SEARCH RESULTS, "SEARCH CATEGORIES" &ORDERED SETS OF VALUES 108 (8) USER SELECTION OF SECOND SEARCH FROM "SEARCH CATEGORIES" Fig.2B.

p.5Flow diagram of the dynamic search category construction routine.

START DYNAMIC SEARCH CATEGORY CONSTRUCTION ROUTINE RECEIVE A REQUEST FOR SEARCH -300 302 304 OBTAIN DATA ENTRIES BASED ON THE SEARCH REQUEST DETERMINE A RANK FOR DATA ENTRIES FOR DISPLAY -306 OBTAIN A SET OF "SEARCH CATEGORIES" AND CATEGORY VALUES -308 DISPLAY DATA ENTRIES IN AN ORDER OF THE DETERMINED RANK 310 FOR A SET OF "SEARCH CATEGORIES,” DISPLAY CATEGORY VALUES IN ORDER CORRESPONDING TO RANK FOR DATA ENTRIES 312 RECEIVE A REQUEST FOR NEXT SEARCH VIA "SEARCH CATEGORY" 314 Fig.3.

p.6Illustrative screen display of search results and categories for "denims, women, pants".

408 XXX.com > Apparel 402 SEARCH Denims, Women, Pants Narrow or Expand Results Showing 1-7 of 110 Results Sort by Relevancy Expand Your Results Remove Keywords: Denims, 1. Women, Pants 412- Narrow by Type Bottoms (53) Active wears (51) Shoes (2) Narrow by Pant Style 414 2. 400 404 GO Buy New $69.99 > Show only 555 Denim Jeans 555 Denim One True Fit Basic Flare Available at Marci 416 418 420 Flare (25) Boot cut (31) Low raise (20) Skinny (15) Slim fit (39) Narrow by Brand 555 Denim (21) Joe and Jill (6) Lady's Jean (28) Pepper (37) Eddie's Jean (16) Calvin Style (3) Narrow by Seller Marci (25) Target (12) Blue (12) Lands' Wear (8) Eddie's shop (20) Mommy's (33) Narrow by Price $50-$99 (25) $100-$199 (53) $0-$49 (35) Joe&Jill Low Raise Flare Jeans 3. 4. Lady' Jean Girls Denim Jeans Buy New $120.00 > Show only Joe&Jill Jeans Available at Target Buy New $70.00 > Show only Lady's Jean Jeans Available at Blue Lady' Jean Boot-Cut Flair Buy New $55.00 > Show only Lady's Jean Jeans Available at Land's Wear : 1-7 of 110 Page: 1 2 3 4 5 6 7 8 Next > 470 460 Fig.4.

p.7Another illustrative screen display for search results with "jeans, flare, black" showing more details.

502 XXX.com > Apparel > Women SEARCH Jeans, Flare, Black 512 514- 516 518 508 Narrow or Expand Results Showing 1-4 of 150 Results Sort by Relevancy Expand Your Results Remove Keywords: Jeans, flare, black 1. Buy New $69.99 GO Narrow by Type Active wears (22) Bottoms (53) tops (48) Narrow by Style Slim Fit (23) Flare (21) Low raise (14) Skinny (14) Boot-cut (32) Narrow by Brand 555 Denim (15) Joe and Jill (20) Eddie's Jean (28) Lady's Jean (8) Pepper (17) Calvin Style (3) Narrow by Seller Marci (5) Blue (12) Target (26) Lands' Wear (18) Eddie's shop (25) Mommy's (3) 555 Denim Slim fit Fashion Jacket Active wear 2. 3. 4. Joe&Jill Wide Flare Jeans Eddie's Jean Girls Denim Low raise Jeans > Show only 555 Denim Jeans Available at Marci Buy New $120.00 > Show only Joe&Jill Jeans Available at Target Buy New $70.00 > Show only Eddie's Jean Jeans Available at Marci 520 Narrow by Price $50-$99 (72) $100-$199 (17) $0-$49 (53) Lady' Jean Boot-Cut Flair : Buy New $55.00 > Show only Lady's Jean Jeans Available at Blue 1-4 of 150 Page: 1 2 3 4 5 6 7 8 Next > Fig.5A.

p.8Illustrative screen display narrowing search results by 'Slim Fit' category.

512 514. 516 518 XXX.com > Apparel > Women>"Jeans, Flare, Black">Slim Fit 500 508 SEARCH Jeans, Flare, Black GO Narrow or Expand Results Showing 1-4 of 23 Results Sort by Relevancy Expand Your Results 1. Remove Keywords: Jeans, flare, black Buy New $69.99 Remove Style Search Category: Slim fit 555 Denim Slim fit Fashion Jacket Active wear > Show only 555 Denim Jeans Available at Marci Narrow by Type Active wears (2) Bottoms (13) tops (8) Narrow by Style Flare (4) Low raise (3) Skinny (14) Narrow by Brand 555 Denim (5) Joe and Jill (14) Eddie's Jean (3) Lands' Wear (1) 2. 3. Eddie's Jean Girls Denim Slim fit Jeans Joe&Jill Slim fit Jacket with Flare detail Buy New $110.00 > Show only Eddie's Jean Jean Available at Blue Buy New $80.00 > Show only Joe&Jill Jeans Available at Blue 520- Narrow by Seller Marci (5) Blue (12) 4. Narrow by Price $50-$99 (5) $100-$199 (10) $0-$49 (8) Lady' Jean Slim fit Active wear Buy New $55.00 > Show only Lady's Jean Jeans Available at Blue : 1-4 of 23 Page: 1 2 3 4 5 6 Next > Fig.5B.

p.9Block diagram of a data entry structure.

ITEM ID 600 ATTRIBUTE FIELD 1 602 ATTRIBUTE FIELD 2 604 ATTRIBUTE FIELD 3 606 100.... ATTRIBUTE FIELD N 608 Fig.6.

p.A system overview diagram showing data flowing between a client and a server.

System overview Client 101 121 Figure 1 Server Data storage 151 111

p.A flowchart diagram describing the transmission of different types of data between a client and a server.

Transmission of different types of data 101 151 Client Server 201 203 Send application data Receive application data 207 205 Receive notification Send notification 209 Take action based on notification 213 211 Receive request for additional information Send request for additional information 215 Gather additional information to fulfill request 217 219 Send additional information Receive additional information Figure 2

p.A flowchart diagram showing how a server receives a request, analyzes data, and returns a result.

Server recieves request, analyzes, returns result 101 151 Client Server 305 301 Detect change in data object Receive identification + data object information 303 307 Send identification + data object information Store data object information 309 313 Analyze data object if necessary Receive data object assessment information 311 Send data object assessment information 315 Process assessment information Figure 3

p.A flowchart diagram illustrating server sending a notification after categorization changes.

Server sends notification after categorization changes Client Server 405 401 Detect change in data object Receive identification + data object information 403 407 Send identification + data object information Store data object information 409 413 Analyze data object if necessary Receive data object assessment information 411 415 Send data object assessment information 417 Process assessment information 421 Wait for change in data object assessment 419 Receive notification 423 Perform notification actions Send notification 425 Send action confirmation Receive notification action confirmation Figure 4 427

p.A flowchart diagram illustrating a server asking for more information after receiving some initial data.

Server asks for more info after receiving some Client Server 505 501 Detect change in data object Receive identification + data object information 503 507 Send identification + data object information Store data object information 509 513 Need additional data object information? No Receive request for additional information about data object ↓ Gather additional information about data 511 Yes 515 Request additional data object information 519 object Receive additional data object information 517 521 Send additional information about data Store additional data object information object 523 527 Analyze data object if necessary Receive data object assessment information 525 529 Send data object assessment information Process assessment information Figure 5

p.A flowchart diagram showing a client sending additional information if the response is unknown.

Client sends additional information if response is unknown Client Server 605 601 Detect change in data object Receive identification + data object information 603 607 Send identification + data object information Receive data item categorization information 613 Store data object information 609 ▼ Analyze data object if necessary 611 Send data object assessment information 615 No Need to send additional data item information? 621 Receive additional data object information 623 Store additional data item information Yes 617 625 Gather additional information about data object Analyze data object if necessary 619 627 Send additional information about data object 629 Send data object assessment information Process assessment information Figure 6

p.A flowchart diagram illustrating a server asking for more information after receiving some, choosing which device to retrieve from.

Server asks for more info after receiving some, choosing which device to retrieve from Client 1 Server 701 705 Detect change in data object Receive identification + data object information 703 709 Send identification + data object information 731 Store data object information 711 Receive data object assessment information Need additional data object information? No 733 Process assessment information Client 2 Receive request for additional information about data object Gather additional information about data Yes 713 Choose device to request data from 717 715 object Request additional data object information 723 719 Receive additional data object 725 Store additional data object information 721 Send additional information about data 727 Analyze data object if necessary 729 ▼ Send data object assessment information' Figure 7 object

p.A flowchart diagram showing file/protocol data analysis on a device.

File/Protocol data analysis on device Gather data being sent or received Identify protocol and track state Known good characteristic Not good Block and/or signal Good Known bad? Bad Good Decision system Not bad Bad

p.A flowchart diagram showing the scan of executable process.

Scan of executable required Known good? Known good Allow Known bad? Known bad Quarantine and/or signal event Bad Good Decision system

p.A diagram depicting gathering data from disparate data sources.

Gathering data from disparate data sources 1003 Web crawler 1005 151 Application marketplace data gathering system Application Marketplace 1011 Network security infrastructure Human-provided information 1007 Data feeds Mobile devices 101 Figure 10

p.A flowchart diagram demonstrating assessment of data object from multiple data sources.

Assessment of data object from multiple data sources. Gather application data for data object 1101 Store application data for data object 1103 Characterize and categorize data object 1109 Store characterization and categorization 1111 Gather device data 1105 Store device data 1107 Perform assessment of data object 1113 Store assessment of data object 1115 Transmit assessment of data object 1117 Figure 11

p.A flowchart diagram combining KG/KB components.

Fig 12. Combined KG/KB components. Client Initiate scan of data object Server 1206 1201 Receive data object information 1202 Determine definition that corresponds to data object 1203 Assessment determined? Yes No 1207 Determine definition that corresponds to data object Assessment determined? 1208 No Yes Transmit data object information to server 1205 1210 Analyze data object information 1211 Receive assessment information Transmit assessment information 1209 1204 Process assessment information Figure 12

p.A block diagram of a system for crawling multiple markets and correlating data.

Developer Web Sites 1360 App Marketplaces 1350 Forums 1355 Clients 1365 Network 1347 Other Download Sites 1362 Collection Server 1310 Analysis Server 1315 Reporting Server 1320 App Binaries 1330 Device Personalities 1346 Metadata 1335 Extracted App Data 1340 Results 1345 Figure 13

p.A top portion of a screen shot of an application that is available on an application marketplace.

Apps News Shopping Gmail Android Market Search Sign in Music Books Movies My Library ANGRY BIRDS Angry Birds Rovio Mobile Ltd TOP DEVELOPER INSTALL More from Developer Angry Birds Rin ROVID MOR LED EDITORS' CHOICE Angry Birds Seasons ROVID MOR (412.002) OVERVIEW Description Use the unique powers of the Angry Birds to destroy the greedy pigs' littresses The survive nithhe: Angry Flais is at sace Cist out revence on the & erdy pags who stole Thar eggs. Vee the unicus powers of each bird to destroy the pigs' icttresses Angry Birds features chalfer-ging physics based yameplay and nouns of replay value. Each of the 300 levels repéres logic, skill, and fouce to solve Terms of use fitto //www.rovio com/eula Privacy Pairy hito www.rovio.com/pnvazy Tweet ABOUT THIS APP RATING Free See more Users who viewed this also viewed Angry Birds Rio Walkthrough Visit Ceveloper's Website mal Geveloper) App Screenshots Air Penguin (20,574 Free Angry Birds Rio Trailer 5000 Grow FCF 爽爽में वो 1,9631 $1.99 Users who installed this also installed Angry Aviary Lite Angry... Rebound 197

p.A bottom portion of the screen shot shown in FIG. 14.

CURRENT VERSION December 2017 REQUIRED OS REQUIRES ANDROID CATEGORY Arcade & astian INSTALLS 5000000 - 100,000,000 SIZE 15M CONTENT RATING Everyone Figure 15

p.An exemplary block diagram of a collection server.

Collection Server 1310 Crawler 1615 App Receiver 1605 Query Generator 1610 Controller 1620 Device Emulator 1625 Data Extractor 1630 Figure 16

p.A flowchart diagram illustrating the steps of an embodiment of the disclosure.

Generate search terms 1710 Compose query based on search term 1715 Submit query 1720 Receive in response a search result identifying an application program 1725 Download and store the application program and associated metadata 1730 Parse metadata for keywords and extract to form search terms for another query 1735 Figure 17

p.A flowchart diagram illustrating the steps of an embodiment of the disclosure.

Select a client personality 1810 Provide client personality to an application source 1815 Receive from source a listing of applications intended for client devices having the client personality 1820 Download the application programs and associated metadata from the source 1825 Figure 18

p.A flowchart diagram illustrating the steps of an embodiment of the disclosure.

Access a source of applications 1910 Request from the source a date-ordered list of applications available at the source 1915 Examine an entry in the list to determine if an application corresponding to the entry has been retrieved previously 1920 If the application was retrieved previously, update an overlap counter variable 1925 Compare the updated overlap counter variable with a threshold overlap value to determine whether a remaining entry in the list should be examined 1930 Based on the comparison, determine that all applications at the source have been retrieved previously and do not examine remaining entries in the list 1935 Based on the comparison, determine that there may be applications at the source that have not been retrieved previously and examine the remaining entry 1940 Figure 19

p.An exemplary block diagram of an analysis server.

Analysis Server 1315 Correlation and Comparison Engine 2010 Inference Engine 2015 Figure 20

p.An exemplary flow diagram illustrating the steps of an embodiment of the disclosure.

Measure similarity between first metadata describing first application and second metadata describing second application 2110 If the similarity is within a threshold degree of similarity, compare the first application with the second application to identify differences between the applications 2115 Identify at least one difference between the first and second application 2120 Based on the identified at least one difference and the degree of similarity, determine that one of the first or second applications is a counterfeit of the other first or second applications 2125 Figure 21

p.An exemplary flow diagram illustrating the steps of an embodiment of the disclosure.

Analyze a first application and generate a first assessment 2210 Correlate a second application with the first application 2215 Based on the first assessment and the correlation, generate a second assessment of the second application 2220 Figure 22

p.3A diagram of a merchant network site connected to various databases and user devices via a network.

MERCHANT NETWORK SITE 100 150 152 154 PROCESSING UNIT CATALOG OF ITEMS TRANSACTION DATABASE GEOGRAPHIC ATTRIBUTE INDEX MEMORY 112 DATABASE MANAGER 110 SEARCH 120 ENGINE 114 DISPLAY 122 COMPONENT 124 TRANSACTION ENGINGE COMPUTER- ACCESSIBLE MEDIA NETWORK INTERFACE 118 126 116 USER USER DEVICE 140 134 NETWORK USER DEVICE USER 130 Fig.1. 132 USER DEVICE USER

p.4A flow diagram of a process for analyzing items and associating geographic attributes.

400 200 RECEIVE ITEM DESCRIPTION AND/OR CONTENT ANALYZE ITEM TO IDENTIFY GEOGRAPHIC ATTRIBUTE DATA STRUCTURE ASSOCIATE IDENTIFIED GEOGRAPHIC ATTRIBUTE WITH ITEM RETURN TO BLOCK 220 240 UNTIL ANALYSIS OF ITEM IS COMPLETE FIG.2. 210 220 150 230 CATALOG OF ITEMS

p.5A flow diagram of a process for searching and identifying items based on a geographic attribute.

312 USER INTERACTION WITH ITEM 314 DIRECT USER INPUT RECEIVE CRITERION SPECIFYING A GEOGRAPHIC ATTRIBUTE 316 USER INTERACTION WITH MAP 310 334 150 USER INTERACTION 332 300 CATALOG OF ITEMS SEARCH CATALOG OF ITEMS FOR ITEMS THAT SHARE A GEOGRAPHIC ATTRIBUTE SPECIFIED BY THE CRITERION MAP 320 IDENTIFY ITEMS LOCATED BY SEARCH FOR DISPLAY 330 RECEIVE IDENTIFICATION OF ITEMS FOR DISPLAY FROM USER 336 RETURN TO BLOCK 310 ENABLE USER TO OBTAIN IDENTIFIED ITEM IN A COMMERCIAL TRANSACTION FIG.3. 340

p.6A diagram of a data structure that can be used to store item information, including geographic attributes.

DATA STRUCTURE 400 410 (TITLE) 420 (AUTHOR) 430 (PUBLISHER) 440 (GENRE) 450 (ISBN) 460 (GEOGRAPHIC ATTRIBUTE) FIG.4.

p.7A screenshot of a web browser showing details of a book item with options to see other items related by geography.

INTERNET BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP ▼ X * X BACK FORWARD STOP REFRESH HOME SEARCH FAVORITES HISTORY MAIL SIZE PRINT >> LINKS ADDRESS HTTP://WWW.ITEMSUPERSTORE.COM 540 NEW SEARCH: 512- THE FIRM BY JOHN GRISHAM 520 522 GO HARDCOVER -- $20.00 NEW ADD TO CART PAPERBACK -- $7.50 USED -- From $2.50 and Up When a Memphis law firm makes Mitch McDeere an offer he really can't refuse, he trades in his old Nissan for a new BMW and put in long hours finding tax shelters. He'd be set for life, if only associates of the firm didn't have a funny habit of dying and the FBI wasn't trying to get him to turn in his colleagues... 524 526 528 New! See other items related by geograohy: Memphis (TN), Cayman Islands, Chicago (IL), Boston (MA), Washington D.C., Panama City (FL), and more. 530 532 534 FIG.5. 536 516 510 518

p.8A screenshot of a web browser showing a map of the southern United States.

INTERNET BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP X ▼ * X BACK FORWARD STOP REFRESH HOME SEARCH FAVORITES HISTORY MAIL SIZE PRINT >> LINKS ADDRESS HTTP://WWW.ITEMSUPERSTORE.COM 616 NEW MAP/SEARCH: United States Midwest 600 GO 612 United States North ZOOM IN 614 ZOOM OUT 610 Nashville Raleigh Oklahoma City Memphis Little Rock Atlanta United States Southwest Dallas- Houston San Antonio Atlantic Ocean New Orleans Jacksonville Gulf of Mexico FIG. 6

p.9A screenshot of a web browser showing a map of Memphis with geographically related items.

INTERNET BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP X ▼ GO 612 ZOOM IN 614 ZOOM OUT 710 BACK FORWARD STOP REFRESH HOME SEARCH FAVORITES HISTORY MAIL SIZE PRINT >> LINKS ADDRESS HTTP://WWW.ITEMSUPERSTORE.COM X ▼ 616 NEW MAP/SEARCH: Millington The Firm Rock 'N Soul Museum Symphonic Elvis Memphis Memphis Charm West Memphis German town Insiders Guide to Memphis Memphis Barbeque Memphis Southaven Horn Lake FIG. 7

p.10A screenshot of a web browser showing a map of downtown Memphis with geographically related items.

INTERNET BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP X ▼ * X 600 BACK FORWARD STOP REFRESH HOME SEARCH FAVORITES HISTORY MAIL SIZE PRINT >>> LINKS ADDRESS HTTP://WWW.ITEMSUPERSTORE.COM GO 612 616 NEW MAP/SEARCH: ZOOM IN 614 ZOOM OUT 810 Dixie:... Greenlaw Park Rock 'N Soul Museum Mississippi River The Pyramid Chez Phillippe Memphis Queen Line Front- Street St. Jude's Childrens Research Hosp. The Firm FIG. 8

p.11A screenshot of a web browser showing a shopping order summary with an option to consider other geographically related items.

920 INTERNET BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP X * X BACK FORWARD STOP REFRESH HOME SEARCH FAVORITES HISTORY MAIL SIZE PRINT >> LINKS ADDRESS HTTP://WWW.ITEMSUPERSTORE.COM GO SHOPPING ORDER SUMMARY 922- The Firm (John Grisham) $20.00 Shipping $3.25 Tax $ 0.00 Total $23.25 924 926 CONTINUE WITH CHECKOUT RETURN TO CATALOG FIG.9. Have you considered other items related to "The Firm" by geography? Memphis (TN) 900 910 930 932 Cayman Islands- 934 Chicago (IL) 936 Boston (MA) 938 Washington D.C.- Panama City (FL) 940 942 more 944

p.12A screenshot of a web browser showing a map with geographic attributes highlighted.

INTERNET BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP X * X BACK FORWARD STOP REFRESH HOME SEARCH FAVORITES HISTORY MAIL SIZE PRINT LINKS ADDRESS HTTP://WWW.ITEMSUPERSTORE.COM NEW MAP/SEARCH: Mexico Chicago 1000 GO Canada Boston ZOOM IN ZOOM OUT 1010 Washington D.C. Memphis- Panama City FIG. 10 Cuba- "Cayman Islands

p.2Diagram showing an example of a video frame with headshots and information pane with actor identification.

100 110 112 102 104 106 Δ HEADSHOT CURATION MODULE 124 COMPUTING DEVICE 120 U.S. Patent Dec. 17, 2024 Sheet 1 of 7 US 12,169,983 B1 ACTOR A NETWORK(S) 118 DATABASE 122 108 ACTOR B FIG. 1 USER DEVICE 114 USER 116

p.3A grid of headshot images with quality scores.

200 202A 0.65 0.95 0.45 0.85 202B 0.95 202C 0.57 202D 0.41 0.78 202E 0.58 0.73 204A 204B 204C 206A 208A 206B 208B 0.72 0.76 0.84 208C 206C 206D 204D 204E 0.91 0.94 206E 0.91 0.51 FIG. 2 208D 208E

p.4A diagram illustrating a filtering process for selecting representative images.

300 302 304 306 308 310 312 314 316 FIG. 3 318

p.5A system diagram showing components for curated headshot generation.

400 PROCESSOR(S) 416 COMPUTER-READABLE MEDIA 418 FEATURE MODULE 420 SOURCE DATA MODULE 424 IMAGES 426 PROCESSOR(S) 406 COMPUTER-READABLE MEDIA 408 DISPLAY 410 CONTENT SERVER 414 USER DEVICE 404 SCORING MODULE 422 ML MODULE 428 USER 402 FIG. 4 NETWORK(S) 412 DATABASE 430

p.6A flowchart describing a process for curating headshots using pose, size, and face costs.

500 502 ACCESS A SET OF IMAGE DATA 504 506 508 510 512 DETERMINE A POSE COST ASSOCIATED WITH FACES IN THE IMAGE DATA DETERMINE A SIZE COST FOR THE IMAGE DATA DETERMINE A FACE COST DETERMINE AN IMAGE OF THE SET OF IMAGE DATA PRESENT THE IMAGE FOR DISPLAY FIG. 5

p.7A flowchart outlining a process for curating headshots using feature values, ranked order, and percentile rank.

600 602 ACCESS A SET OF IMAGE DATA 604 614 SELECT AN IMAGE OF THE SET OF IMAGES DETERMINE A FEATURE VALUE FOR THE SET OF IMAGES 606 DETERMINE A RANKED ORDER FOR THE SET OF IMAGES 608 DETERMINE A PERCENTILE RANK FOR THE SET OF IMAGES 610 REMOVE ONE OR MORE IMAGES FROM THE SET OF IMAGES 612 PRESENT THE IMAGE FOR DISPLAY FIG. 6

p.8A block diagram of a computing system used for generating and curating headshots.

U.S. Patent Dec. 17, 2024 Sheet 7 of 7 US 12,169,983 B1 COMPUTING DEVICE(S) 700 MEMORY DEVICE(S) MODULES DATA STORE PROCESSOR(S) 712 710 702 I/O DEVICES 704 NETWORKING DEVICES 706 FIG. 7 708 714

p.9This is a flowchart showing an example process for providing recommended keywords in response to an image query.

Receive an image query from a computing device ↓ Perform search of datastore of images for images matching the image query ↑ Obtain keywords associated with the matched content Generate recommended keyword search strings based on obtained keywords ↓ Provide recommended keyword search strings to user 712 Search string selected? Provide content associated with images matching image query Receive a selected keyword search string from the computing device Perform search of datastore of content using selected search string Provide identified content in response to selected search string to user

p.3This is a diagram showing the flow of providing inline search suggestions.

BRAND ABC HANDBAG A 119 RESULTS FOR "BRAND ABC HANDBAG" <ITEM-1 ITEM-2 ITEM-3> <ITEM-4 ITEM-5 ITEM-6> SEGMENT LIST SEGMENT-1 BRAND ABC SEGMENT-2 HANDBAG DATA 112 114 100 RECEIVE A SEARCH STRING FROM A USER 102 IDENTIFY SEGMENTS OF THE SEARCH SRING 104 SEGMENT-1 TERM SUBSTITUTIONS: 10 RE-FORMULATIONS: 7 TERM SWAP: 3 SEGMENT-2 TERM SUBSTITUTIONS: 7 RE-FORMULATIONS: 5 TERM SWAP: 3 ANALYZE SEARCH STRINGS SUBMITTED BY USERS 106 BRAND ABC HANDBAG BRAND 123 HANDBAG BRAND XYZ HANDBAG LEATHER HANDBAG 110 PROVIDE SUGGESTED SEARCH TERMS FOR IDENTIFIED SEGMENT TO THE USER 108 FIG. 1

p.6The image shows a screenshot of a web browser interface with inline search suggestions.

WEB BROWSER EILE EDIT VIEW EAVORITES TOOLS HELP ADDRESS WWW.MYCOMPANY.COM 400 GO WWW.MYCOMPANY.COM 402✓ BRAND ABC HANDBAG SEARCH BRAND 123 RED HANDBAG BRAND XYZ VELVET HANDBAG BRAND ABC SUEDE HANDBAG BRAND ABC LEATHER HANDBAG 119 RESULTS FOR "BRAND ABC HANDBAG" ITEM-1 ITEM-2 ITEM-3 ITEM-4 ITEM-5 ITEM-6 ITEM-7 ITEM-8

p.7An image of another interface mockup displaying a search suggestion feature.

WEB BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP ADDRESS WWW.MYCOMPANY.COM 500 GO WWW.MYCOMPANY.COM BRAND ABC HANDBAG SEARCH 502 MEN'S BRAND ABC HANDBAG LARGE BRAND ABC HANDBAG WOMEN'S BRAND ABC HANDBAG LEATHER BRAND ABC HANDBAG 119 RESULTS FOR "BRAND ABC HANDBAG" ITEM-1 ITEM-2 ITEM-3 ITEM-4 ITEM-5 ITEM-6 ITEM-7 ITEM-8 FIG. 5

p.8An image depicting another interface mockup with search bar suggestions.

A WEB BROWSER FILE EDIT VIEW FAVORITES TOOLS HELP ADDRESS WWW.MYCOMPANY.COM WWW.MYCOMPANY.COM 602 GO BRAND ABC HANDBAG SEARCH BRAND 123 HANDBAG WEEKEND BRAND XYZ HANDBAG TOTE BRAND ABC HANDBAG CLUTCH BRAND ABC HANDBAG LEATHER CLUTCH 119 RESULTS FOR "BRAND ABC HANDBAG" ITEM-1 ITEM-2 ITEM-3 ITEM-4 ITEM-5 ITEM-6 ITEM-7 ITEM-8 FIG. 6 600

p.9A flowchart outlines steps to provide inline search suggestions by a user.

RECEIVE A SEARCH STRING SUBMITTED BY A USER 702 IDENTIFY SEGMENTS OF THE SEARCH STRING 704 ANALYZE SEARCH STRINGS SUBMITTED BY USERS OF THE ELECTRONIC MARKETPLACE 706 DETERMINE A PLURALITY OF SUGGESTED SEARCH TERMS BASED ON THE ANALYSIS 708 RECEIVE A SELECTION OF A PARTICULAR SEGMENT FROM THE USER 710 PROVIDE SUGGESTED SEARCH TERMS RELATED TO THE PARTICULAR SEGMENT TO THE USER 712 FIG. 7 700

p.10A flow diagram for inserting search terms based on user input is shown.

RECEIVE A SEARCH STRING SUBMITTED BY A USER 802 IDENTIFY AN ACTION PERFORMED BY THE USER IN RELATION TO THE SEARCH STRING 804 IDENTIFY SEGMENTS OF THE SEARCH STRING 806 RECEIVE AN INPUT FROM THE USER 808 INSERT SUGGESTED SEARCH TERMS INTO THE SEARCH STRING BASED AT LEAST IN PART ON THE INPUT 810 IDENTIFY NEW SEGMENTS IN THE SEARCH STRING 812 STORE THE NEW SEGMENTS IN A DATA STORE 814 FIG. 8 800

p.11A network environment diagram, with client device, web server, application server, and datastore.

CLIENT DEVICE 902 NETWORK(S) 904 000 906 WEB SERVER APPLICATION SERVER 908 PRODUCTION 912 TESTING 914 USER INFORMATION 916 DATA STORE 910 FIG. 9

p.3This is a block diagram illustrating the client computer system.

CPU 110 client computer system 100 input/output device storage device 121 130 memory query server 131 computer-readable media drive 122 query result ranking facility 132 network connection 123 item rating tables 133 FIG. 1 120

p.4This is a flow diagram illustrating the steps taken to generate a new rating table.

begin 201 initialize rating table for new rating period 202 identify item selections from query results during period and terms of queries for each item selection from query result during period item terms in query next item selection 203 204 208 end FIG. 2 205 for each term in query increase rating score for (term, item) next term 206 207

p.5This is an item rating table diagram.

item rating table 300 term item identifer score : dynamics 0801062272 1 301 dynamics 1883823064 22 302 dynamics 9676530409 7 303 : human 0814403484 16 304 human 1883823064 45 305 human 6303702473 3 306 : FIG. 3

p.6This is a category/item rating table diagram.

category/item rating table 3A00 category term item identifier score : science dynamics 0801062272 1 3A01 science dynamics 9676530409 7 3A02 : science human 6303702473 3 3A03 : : : social science dynamics 1883823064 23 3A04 : : social science human 1883823064 46 3A05 social science human 0814403484 16 3A06 : : FIG. 3A

p.7This is another item rating table diagram.

item rating table 400 item term score identifer dynamics 0801062272 1 401 dynamics 1883823064 23 402 dynamics 9676530409 7 403 : human 0814403484 16 404 human 1883823064 46 405 human 6303702473 3 406 FIG. 4

p.8This is a diagram illustrating how composite periods are built up.

constituent period constituent period item rating table composite period 8-Feb-98 9-Feb-98 10-Feb-98 11-Feb-98 501 502 503 12-Feb-98 504 8-Feb-98 to 12-Feb-98 13-Feb-98 505 506 9-Feb-98 to 13-Feb-98 item rating tables 500 composite period item rating table 515 516 FIG. 5

p.9This is a rating table for a composite period diagram.

item rating table 600 term item identifer score : dynamics 0801062272 4 601 dynamics 1883823064 116 602 dynamics 1887650024 2 607 -dynamics 9676530409 45 603 : human 0814403484 77 604 human 1883823064 211 605 human 6303702473 12 606 FIG. 6

p.10This is a flow diagram illustrating identifying user selections within a web server log.

begin position pointer 1 at top of log 701 702 repeat traverse forward with pointer 1 to next item selection event 703 from item selection event, extract item selected and session 704 identifier synchronize position of pointer 2 with that of pointer 1 705 traverse backwards with pointer 2 to query event having matching session identifier 706 from query event, extract terms of query 707 until end of log is reached with pointer 1 end 708 FIG. 7

p.11This is a flow diagram illustrating ordering a query result using a rating table by generating a ranking value for each item in the query result.

begin for each item identified in query result 801 802 initialize ranking value for item 806 combine scores for item to generate ranking value for item 807 next item display items identified in query result in accordance with ranking values end 808 803 for each term in query 804 determine ranking score for (term, item) in rating table 805 FIG. 8 next term

p.12This is a flow diagram showing the steps preferably performed by the facility to select a few items in a query result having the highest ranking values using a rating table.

begin 901 for each term in query identify table entries matching term having 3 highest rating scores 902 903 next term for each unique item among identified entries 904 combine all scores for item among identified entries 905 906 next item select for prominent display items having top 3 combined scores end FIG. 9 907

p.3This figure illustrates a clustered representation of a user's purchase history or other item collection.

C1 C4 C2 C3 C5 60

p.4This figure illustrates a process for selecting recommendation source items from the user's item collection.

RETRIEVE ITEM COLLECTION LIST FOR TARGET USER 30 -32 IS COLLECTION LARGE ENOUGH TO APPLY CLUSTERING ? YES APPLY CLUSTERING ALGORITHM -36 NO USE ALL ITEMS IN COLLECTION AS SOURCES, OR SELECT SOURCES BASED ON NON-CLUSTER-BASED CRITERIA -34 -38 ANALYZE CLUSTERING, OPTIONALLY IN COMBINATION WITH OTHER USER DATA, TO SELECT ITEMS TO INCLUDE OR EXCLUDE AS SOURCES GENERATE RECOMMENDATIONS BASED ON SOURCE ITEMS 40 OUTPUT RECOMMENDATIONS TO USER 42

p.6This figure illustrates a cluster-based process for filtering the output of a recommendation engine.

ITEM COLLECTION OF TARGET USER 60 CLUSTERING ENGINE "NOT INTERESTED" ITEM CLUSTERS CLUSTER-BASED FILTERING FILTERED RECOMMENDATIONS -64 SOURCE ITEMS FOR TARGET USER 62 RECOMMENDATION ENGINE ITEM RECOMMENDATIONS

p.7This figure illustrates processes for organizing a set of recommended items into cluster-based categories for presentation to users.

GENERATE RECOMMENDATIONS APPLY CLUSTERING ALGORITHM TO SET OF RECOMMENDED ITEMS 70 72 ASSIGN A CATEGORY NAME TO EACH CLUSTER BASED ON ATTRIBUTES (e.g., SUBJECT KEYWORDS OR BROWSE CATEGORIES) OF THE ITEMS IN THAT CLUSTER -74 DISPLAY RECOMMENDATIONS ARRANGED BY CATEGORY NAME, OPTIONALLY USING A "BROWSE CLOUD" UI 76 APPLY CLUSTERING TO ITEM COLLECTION OF TARGET USER -80 OPTIONALLY ANALYZE ITEM CLUSTERS TO IDENTIFY AND EXCLUDE ANY CLUSTERS THAT LIKELY DO NOT REPRESENT AN INTEREST OF THE TARGET USER 82 ASSIGN A CATEGORY NAME TO EACH NON-EXCLUDED CLUSTER -84 GENERATE RECOMMENDATIONS FOR TARGET USER 86 ASSIGN EACH RECOMMENDED ITEM (WHERE POSSIBLE) TO ONE OF TARGET USER'S INTEREST CLUSTERS 88 DISPLAY RECOMMENDATIONS ARRANGED BY CATEGORY NAME 90

p.9This figure illustrates a portion of a web page showing an example browse cloud interface that may be used to organize a set of recommended items into cluster-based categories.

Life of Brian 1DVD) Dreamland (ABS CD) Chung Cup and του ώνα of Tae Kwon Do (Paperback) Mike Multown and His Steam Shovel Hardcover) Sud Wears Kabbalah For Viomen (Hardcover) Classics Comedy Cook, Robin Fantasy Hard-Boded AMIRISLION fedtime & Creaming Action & Adventure Humorous International Kabbalah L'angle, Madeleine Latin Music Literary Mystery Picture Books Pop Rock Readers Retro Swing ສົມກning & Ingging Science Fiction Science Fiction, Fantasy, & Magic Suspense Thrillers United States

p.10This figure illustrates an example hierarchical browse structure.

N2 N3 N6 N7 II IN 12 N12 N4 N5 N8 N9 N10 IIN 16 N13 13 14 15 17 18

p.11This figure illustrates one example how the various cluster-related features may be implemented in the context of a web site that hosts an electronic catalog.

102 86 ELECTRONIC CATALOG SYSTEM (WEB SITE) 100 62 INTERNET WEB SERVERS 102 102 TEMPLATES 104 106 CATALOG SERVICE ITEM CONTENT AND BROWSE STRUCTURE 108 110 RECOMMENDATION ENGINE RECOMMENDATIONS SERVICE SIMILAR ITEMS TABLE(S) ITEM-TO-ITEM MAPPINGS 112 120 CLUSTER GENERATION AND ANALYSIS USER DATA • PURCHASE HISTORIES • RECENTLY VIEWED ITEMS • ITEM RATINGS • ITEM TAGS 116 118 ITEM-ASSOCIATION MINING (OFF-LINE)

p.4This diagram shows a flowchart of the keyword combination scoring routine.

START KEYWORD COMBINATION SCORING ROUTINE OBTAIN A SET OF SEARCH QUERY KEYWORDS DETERMINE SET OF KEYWORD FIELD COMBINATIONS SELECT KEYWORD UNIQUE COMBINATION CALCULATE SCORE FOR THE SELECTED COMBINATION YES ANY REMAINING UNIQUE COMBINATIONS NO IDENTIFY COMBINATION WITH OPTIMAL SCORE END

p.5This diagram shows a matrix for associating search query keywords to terms in a set of characterization fields.

T1 T2 T3 TITLE AUTHOR TEXT

p.5This block diagram illustrates a set of potential keyword field combinations for a search query.

TITLE {T1}, {T2}, {T1, T2}, {NULL} AUTHOR {T2}, {T3}, {T2, T3}, {NULL} TEXT {T1}, {T2}, {T3}, {T1, T2}, {T2, T3}, {T1, T3}, {T1, T2, T3}, {NULL}

p.1This is a system diagram for generating knowledge graphs.

NETWORK(S) 130 USERS 106 USER DEVICES 108 CHATBOT INTERFACE(S) 114 ARTIFICIAL INTELLIGENCE (AI) SYSTEM 116 LARGE LANGUAGE MODELS (LLMS) 118 SERVICE PROVIDER SYSTEM 102 SERVICE PROVIDER

p.3This figure is a flow diagram illustrating how a natural language query is processed.

NATURAL-LANGUAGE QUERY 126 SELECT ?president ?birthdate WHERE { ?president rdf:type dbo:President. ?president dbo:birthDate ?birthDate. FILTER (year(?birthDate) => 1946 && year (?birthDate) <= 1964) } ?president dbo:nationality dbr:United_States

p.5This diagram illustrates the feedback and provenance user interface for the system.

Report Highlighted Errors 410

p.9This is a flowchart of a process where a query is received and a large language model is used to determine an answer.

OBTAIN A QUERY THAT IS USABLE TO RETRIEVE AN ANSWER FOR THE QUERY FROM THE KNOWLEDGE GRAPH 702 DETERMINE THAT THE ANSWER TO THE QUERY IS NOT INCLUDED IN THE KNOWLEDGE GRAPH 704 PROMPT A LARGE LANGUAGE MODEL (LLM) TO DETERMINE THE ANSWER TO THE QUERY BASED AT LEAST IN PART ON THE ANSWER NOT BEING INCLUDED IN THE KNOWLEDGE GRAPH 706 RECEIVE, AS AN OUTPUT FROM THE LLM, THE ANSWER TO THE QUERY 708 ADD THE ANSWER TO THE QUERY TO THE KNOWLEDGE GRAPH 710

p.15This image displays the dataflow and communication between components in the system, showing user inputs, data centers, networks, and system 104.

DATA CENTER 1204A DATA CENTER 1204B NETWORK(S) 130 KNOWLEDGE-GRAPH SYSTEM 104 DATA CENTER 1204C USERS 106 SERVICE PROVIDER NETWORK 102 DATA CENTER 1204N

p.1This diagram illustrates a computing environment for determining actions related to task items using machine learning.

THIRD PARTY ENTITIES 114 PAST USERS 112 NETWORK(S) 128 NOTIFICATION 126 LIST 122 LIST ITEM 120(1) SERVICE PROVIDER 102 ENGINE 104 TASK PERFORMANCE APPLICATION 106 TASK PERFORMANCE MODEL 108 : LIST ITEM 120(N) FIG. 1

p.2This block diagram depicts an illustrative computing architecture for determining actions related to task items via machine learning.

PROCESSOR(S) 202 COMPUTER-READABLE MEDIA 204 DATA COLLECTION MODULE 206 MACHINE LEARNING ENGINE 104 TASK PERFORMANCE APPLICATION 106 TASK PERFORMANCE MODEL 108 THRESHOLD PREDICTION MODEL 208 DEPENDENCY PREDICTION MODEL 210 INTENT PREDICTION MODEL 212 USER PROFILE DATA 214 TASK ITEM DATA 216 MODEL DATA 218 FIG. 2

p.3This flow diagram illustrates a process for determining actions to perform on a task item using machine learning.

IDENTIFY SEGMENT POPULATION 302 IDENTIFY ITEM RELATED DATA 304 GENERATE DATA MODEL USING MACHINE LEARNING TO RECOMMEND ACTIONS FOR CAUSING USER ACTIONS WITH RESPECT TO LIST ITEMS 306 DEPLOY MODEL 308 FIG. 3

p.4This flow diagram illustrates a process for determining a threshold for user action related to a task item, using machine learning.

IDENTIFY SEGMENT POPULATION 402 DETERMINE ITEM RELATED DATA 404 GENERATE DATA MODEL USING MACHINE LEARNING TO DETERMINE THRESHOLD VALUE FOR LIST ITEM 406 DEPLOY MODEL 408 FIG. 4

p.5This flow diagram illustrates a process for mining an action to perform with respect to a task item using machine learning.

ACCESS HISTORICAL USER PROFILE DATA AND HISTORICAL TASK ITEM DATA 502 ANALYZE, USING ML, THE HISTORICAL USER PROFILE DATA TO DETERMINE A FIRST PLURALITY OF DATA INDICATORS THAT CORRELATE WITH PERFORMANCE OF A TASK ITEM 504 ANALYZE, USING ML, THE HISTORICAL TASK ITEM DATA TO DETERMINE A SECOND PLURALITY OF DATA INDICATORS THAT CORRELATE PERFORMANCE OF A TASK ITEM 506 CREATE, USING ML, A FUNCTION USING THE INDICATORS, THE FUNCTION TO RECOMMEND AN ACTION FOR INFLUENCING A USER TO PERFORM A TASK ITEM 508 DETERMINE WEIGHTS FOR INDIVIDUAL INDICATORS THAT APPORTION THE INFLUENCE OF THE INDICATORS IN DETERMINING THE RECOMMENDED ACTION 510 DEPLOY FUNCTION WITH WEIGHTS 512 NO UPDATE MODEL? 514 YES FIG. 5

p.6This is an example of interface for applying graphical effects to a listing of a task item.

TO DO LIST ORDER WEDDING GIFT FOR AUNT FOLLOW UP WITH CUSTOMER SERVICE

p.7This is an example of an interface for managing task items.

TO DO LIST RETURN PHONE CALL LINDA RSVP FOR B-DAY PARTY PURCHASE B-DAY GIFT EMAIL THANK YOU NOTE DISPOSE RECYCLABLE WASTE TO DO LIST EMAIL THANK YOU NOTE RENEW CAR INSURANCE RETURN PHONE CALL LINDA COMPLETED RSVP FOR B-DAY PARTY PURCHASE B-DAY GIFT

p.2This diagram shows the process of receiving a query, determining search results and providing the search results to a user.

100 Receive, from a user computing device, a query comprising a search string 102 Determine, based on the query, a query vector 112 Compare the query vector to representative vectors 116 Compare the query vector to candidate item vectors 124 Determine result items and transmit result items to the user computing device 134 ... ... 106 110 108 104 110 ►{Q1, Q2, Q3} 114 118 {A1, A2, A3} 120 ... {Q1, Q2, Q3} 114 {B1, B2, B3} 122 {C1, C2, C3} 120 {B1, B2, B3} 126 {b1-1, b1-2, b1-3} ... {Q1, Q2, Q3} 128 {b2-1, b2-2, b2-3} 114 130 {b3-1, b3-2, b3-3 132 {b4-1, b4-2, b4-3 ... FIG. 1 110 136 108 104

p.3This diagram illustrates the example process for determining sparse vector representation of items in an online catalog.

200 202 204 Semantic analysis component 210 206 208 {brand, style, color, size, , etc.} {brand, style, color, size, .., etc.} {brand, style, color, size,..., etc.} 212 Vector representation component 214 {b2-1, b2-2, b2-3} {b3-1, b3-2, b3-3} 216 222 Sparse vector generation component 218 {b1-1, null, null, b1-2, null, null, b1-3, null} 220 {full, b2-1, b2-2, null, null, null, b2-3, null} 224 FIG. 2 {null, null, b3-1, null, b3-2, null, null, b3-3}

p.4This diagram illustrates an example process for determining clusters of items and partitions within a cluster based on vector representations of the items.

300 Generate similarity graph 302 Identify representative items 306 Generate clusters 312 Partition clusters 314 * * ... : * * * * * * 304 FIG. 3 * * : ... : *: **** 308 308 310 * 310 * ***:* : 1.[***] 2.[* * * 308 * 3.[***] n. [* 1. [* * 2.[* * 310 n. [*

p.5This diagram shows the comparison of a vector representation of a query and clusters of items.

400 1. Q1 2. null 3. null Query 402 4. Q4 5. Q5 6. null Cluster A 404 1. [A1, a1, a2, a4] 2. [a1, a3] 3. [a2] 4. [A1, a4] 5. [A1, a1, a2, a4] 6. [A1, a1, a2] Similarity comparison component 408 FIG. 4 Cluster B 406 1. [b2] 2. [B1, b1, b2, b4] 3. [B1, b2, b3] 4. [b2, b3, b4] 5. [b1, b2] יוווין 6. [B1, b1, b2, b3, b4]

p.6This diagram shows a flow chart to sending search results to a user based on a query and a clustered catalog.

500 Determine a plurality of items 502 Generate a similarity graph based at least in part on distances between vector representations of the plurality of items in a vector space 504 Determine, based at least in part on the similarity graph, an item of the plurality of items as a representative item for a cluster of items 506 Receive a user query 508 Generate, based at least in part on the user query, a query vector that is a first vector representation of the user query 510 Compare the query vector with an item vector that is a second vector representation of the item 512 Determine, based at least in part on the comparing the query vector with the item vector, the cluster of items as a candidate cluster 514 Determine a candidate item in the cluster of items 516 Send, to a computing device based at least in part on the user query, search results comprising the candidate item 518 FIG. 5

p.7This diagram is a block diagram of the example architecture for data processing.

Processing Element 602 Display Component 610 Input Device 612 600 Storage Element 604 Operating System Transfer App 608 Power Supply 614 Sensor 628 606 Communication I/F 616 SR I/F 620 Wireless 618 GPS 624 Mobile I/F 622 Wired 626 FIG. 6

p.8This diagram shows an example system for sending and providing data.

700 706a 704a Data Center 702 710a 712a- Server 712b VM RSVM 716a- Manager Server 712d Network 708 720 718 714 Gateway Router Server Manager 710b 712c- VM 716b- 706b 704b FIG. 7 RSVM Manager

p.3This is a block diagram illustrating the electronic catalog system.

BLURB AUTHORING PIPELINE BLOGS (AUTHOR-SPECIFIC) INTERNAL BLURBS EXTERNAL SOURCES BLURBS INTERNAL BLURBS EXTERNAL BLURBS PERSONALIZED LOGS (PLOGS) SELECTED BLURBS BLURB DATA PERSONALIZED BLURB SELECTION ITEM SELECTION INTERNET HISTORIES, AUTHOR REPUTATIONS, etc. CATALOG TRANSACTIONS BLURB VOTING USER PROFILES • ITEM SELECTION HISTORIES • AUTHOR VOTES/REPUTATIONS ITEM DESCRIPTIONS : AUTHOR VOTES BLURB VOTES

p.4This is a blurb authoring page.

Write Your Blurb Welcome, Jane Doe. (If you are not Jane Doe, click here) Enter an informative headline and write your blurb below. Use the editing tools to customize your text, add links to websites, or easily add links to items in our catalog. You can highlight text in the editor window and click “add item link, or click “add item link” without highlighting anything to see other options. Headline: BI Add Web link Add item link HELP Preview

p.5This is a preview of the blurb authoring page.

Preview Your Blurb Your Blurb will appear as seen below. If you want to make changes, then click the back button. If you are satisfied with your blurb click continue. What is nanotechnology? Nanotechnology is the quest to build machinery of extremely COVER small size as seen in the book Prey, on the order of 100 ART nanometers, or a hundred billionth of a meter. Such machines would be about 1,000 times smaller than the the diameter of a human hair. Pundits predict these tiny machines will provide everything from miniaturized components to new cancer treatments to new weapons of war.

p.6This is a personalized Web log.

Your Plog a personalized web log An internal blurb by Jane Doe What is nanotechnology? Nanotechnology is the quest to build machinery of extremely small size as seen in the book Prey, on the order of 100 Rate it Discuss Write a blurb More options 3/12/03 10:15am An external blurb from Elizabeth, a Coffee Connoisseur The Best Cafes in San Francisco.

p.7This is a page showing the new Plog entries.

New Plog entries View your Plog Page The best C++ books by Bill Jones After years of writing C++ programs, I've finally found two books... What is nanotechnology by Jane Doe Nanotechnology is the quest to build...

p.8This diagram illustrates the Personalized Blurb Selection architecture.

PERSONALIZED BLURB SELECTION SELECTION ALGORITHM 1 SELECTION ALGORITHM 2 ... SELECTION ALGORITHM N SELECTED BLURBS FOR TARGET USER ARBITER BLURB DATA USER PROFILE DATA USER FEEDBACK BLURB NOMINATIONS AND SCORES

p.9This is a flow chart explaining the implicit voting method.

PRESENT CONTENT ITEM TO USERS WITH OPTION TO IMPLICITLY AND/OR EXPLICITLY VOTE MONITOR IMPLICIT AND/OR EXPLICIT VOTES AND ASSOCIATE ITEM SELECTION HISTORIES OF VOTERS WITH CONTENT ITEM FOR EACH TARGET USER, CALCULATE DEGREE TO WHICH TARGET USER'S ITEM SELECTION HISTORY IS SIMILAR TO ITEM SELECTION HISTORIES OF THOSE WHO VOTED FAVORABLY (AND OPTIONALLY, IS DISSIMILAR TO THOSE WHO VOTED UNFAVORABLY) DETERMINE WHETHER TO PRESENT CONTENT ITEM TO TARGET USER BASED ON CALCULATED DEGREE OF SIMILARITY/ DISSIMILARITY FOR THAT TARGET USER, AND POSSIBLY OTHER CRITERIA

p.10This is a diagram illustrating the web-based catalog system.

WEB SERVER TEMPLATE PROCESSOR CATALOG SERVICE APP BLURBS SERVICE APP USER ACCOUNTS SERVICE APP WEB-BASED CATALOG SYSTEM ITEM DESCRIPTIONS WEB PAGE TEMPLATES (OTHER SERVICES) BLURBS USER PROFILES RSS FEED EXTERNAL SOURCES

p.2A state diagram illustrating a universal query routine for obtaining search results from multiple different search indexes.

BOOKS DVD MUSIC SOFTWARE (2) SUBMIT & OBTAIN SEARCH INDEX RESULT SETS QUERY CONTROLLER 113 100 111 APPAREL (3) COMPUTE QUERY INDEX ASSOCIATIONS (4) DETERMINE HIGHEST PROBABILITY MATCHES (5) ADD MATCHES TO UNIVERSAL QUERY RESULT SET (1) SUBMIT QUERY (E.G., “HOWARD") (6) RETURN UNIVERSAL QUERY RESULT SET 10 115 Fig.1.

p.3A block diagram of the search index result sets obtained in response to a query.

QUEUED SEARCH INDEX RESULT SETS RANK SCORE ITEM 1 100% Howard, the Autobiography 2 99% Howard's Evil Empire 3 95% Howard and Eugene Travel the World 4 92% Howard on Math 5 85% Howie and Emma RANK SCORE ITEM 1 99% Howard and God 2 95% Howard Visits Rome 3 92% Howard and His Bike 4 90% Howie Goes Home 5 80% Howie Drives the Bus RANK SCORE ITEM 1 4001 Howard Sings Christmas Carols 2 3675 Howard's Greatest Hits 3 3321 Howard, the Big Green Man 4 2500 Howard, All Around the World 5 1987 Harold Sings RANK SCORE ITEM 1 100 Howard on Computers 2 77 Howard Teaches 3 65 Helping Howard Read 4 43 Helping Howard Spell 5 41 Helping Howard (Three Pack Collection) RANK SCORE ITEM 1 90% Howard Brand Overalls 2 88% Howard Headware 3 70% Hard Hats 4 65% Hot Socks 5 63% Hobbie Horse Shoes

p.4A graphical representation of a universal query result set generated by combining the identification of items from the different search index result sets returned in response to a query.

RANK 1 2 3 4 5 6 7 8 9 10 QUERY RESULTS ITEM Howard, the Autobiography Howard and God Howard Visits Rome Howard's Evil Empire Howard on Computers Howard and Eugene Travel the World Howard and His Bike Howard Sings Christmas Carols Howard Brand Overalls Howard Teaches

p.5A flow diagram of a universal query routine.

UNIVERSAL QUERY ROUTINE 401 RECEIVE QUERY 400 DETERMINE SEARCH INDEXES SUBMIT QUERY TO DETERMINED SEARCH INDEXES QUEUE RESULTS FROM DETERMINED INDEXES (SEARCH INDEX RESULT SETS) COMPUTE QUERY INDEX ASSOCIATION COMPUTE ALLOCATION SCORE SUBROUTINE COMPUTE UNIVERSAL ITEM SCORE SUBROUTINE COMBINE ALLOCATION SCORES AND ITEM SCORES FOR EACH TOP ORDER ITEM ADD TOP ORDER ITEM WITH HIGHEST COMBINED SCORE ADDITIONAL RESULTS NEEDED? YES NO RETURN UNIVERSAL QUERY RESULT SET END

p.6A flow diagram of the query index association subroutine.

QUERY INDEX ASSOCIATION SUBROUTINE YES RETRIEVE QUERY INDEX ASSOCIATION QUERY INDEX ASSOCIATION EXIST? NO IDENTIFY SIMILAR HISTORIC QUERIES GENERATE APPROPRIATENESS SCORES FOR EACH SEARCH INDEX BASED ON HISTORIC QUERIES APPLY WEIGHTING TO APPROPRIATENESS SCORES COMPUTE QUERY INDEX ASSOCIATION FOR DETERMINED SEARCH INDEXES RETURN QUERY INDEX ASSOCIATIONS

p.7A flow diagram of the allocation score subroutine.

ALLOCATION SCORE SUBROUTINE DETERMINE COMPOSITION OF ITEMS IN UNIVERSAL QUERY RESULT SET COMPUTE ALLOCATION SCORE FOR DETERMINED SEARCH INDEX ADDITIONAL DETERMINED SEARCH INDEX? YES NO RETURN ALLOCATION SCORES

p.8A flow diagram of a universal item score subroutine.

UNIVERSAL ITEM SCORE SUBROUTINE DETERMINE RELEVANCE SCORE FOR SEARCH INDEX RESULT SET COMPUTE ITEM SCORE FOR NEXT TOP-ORDER ITEM IN SEARCH INDEX RESULT SET ADDITIONAL YES SEARCH INDEX RESULT SETS? NO RETURN UNIVERSAL ITEM SCORES

p.2A diagram illustrating a multi-objective ranking system in an online retail context.

Online Retailer 114 Objective(s) 122 112 104 Training Data 120 102 Multi-objective ranking system 102 Primary Objective (Purchase) + Secondary objectives Primary Objective (Purchase) Ranking 130 Ranking 140 Pur- Non- Pur- Non- Pur- Non- chase Quality defect chase Quality defect chase Quality defect 110 Item 1 0 0 0 Item 2 1 0 1 Item 2 1 0 1 Item 2 1 0 1 Item 1 0 0 0 Item 3 0 1 1 Item 3 0 1 1 Item 3 0 1 1 Item 1 0 0 0 INDCG 0.63 0.5 0.69 NDCG 1 0.5 0.92 1 Purchase NDCG optimal 0.63 Purchase NDCG Optimal, Non-defect NDCG optimal, Quality NDCG improved 1 FIG. 1

p.3A diagram presenting the optimization problem and Lagrangian function related to the multi-objective ranking system.

Online Retailer 114 Objective(s) 122 112 102 Multi-objective ranking system 102 104 210 min Cpm (s) s.t. Ct (s) ≤ b, t = 1,...,T 212 110 L (s, a) = 3 =Cm (s) + Σα² (Ct (s) -b²) 214 T Lk (s, a) = Cpm (s) + Σα² (0 (৪) (8)-b²) 1 2μκ -1)2 FIG. 2

p.4Pseudocode illustrating the integration of Augmented Lagrangian into the LambdaMART ranking algorithm.

Input: Number of trees N, number of leaves per tree L, learning rate n. AL parameter μ'. Initial Lagrange multiplier estimate of = 0, t = 1, .., T. Given initial BaseModel foreach q ∈ Q do | fo(x) = Base Model(x).i∈I /* If BaseModel is empty, set fo(x) = 0 end for n=1 to N do foreach q∈Q, i el do end {R} * * = xPma + 1 1 L /* Create an L leaf tree on {(x,y)GEQ.iel Ral Σ /* Assign leaf values on Newton step estimate */ foreach q ∈ Q, ie I do fn(x) = fn-1(x1) +η Σιιδ (2) € Rmi) /* Take step with learning rate for t=1 to T do Compute cost Ct (s) with {s} = f (x)}qEQJEl9 {1=fn(x)}qEQA Update of via Eq. 18 or Eq. 19 end end end FIG. 3

p.5A block diagram showing an example architecture of a computing device.

Processing Element 404 Display Component 406 Input Device 408 Power Supply 414 Sensor 430 400 Storage Element 402 Operating System 422 Transfer App 424 Communication I/F 412 SR I/F 434 GPS 438 Wireless 436 Mobile I/F 440 Π Microphone 470 Wired 442 FIG. 4

p.6A flow diagram illustrating an example process for multi-objective ranking of search results.

502 500 Determining a first ranking for a plurality of search results using a first machine learning model optimized for a first objective for ranking search results 504 Determining a second objective for ranking search results 506 508 Determining a constraint for the second objective updating the first machine learning model to generate an updated first machine learning model by minimizing a cost of the first objective subject to the constraint, wherein violations of the constraint are penalized using a penalty term a 510 Determining a second ranking for the plurality of search results using the updated first machine learning model, wherein the search results of the second ranking are reordered relative to the search results of the first ranking FIG. 5

p.7A diagram illustrating a system for sending and providing data.

62a 60a Data Center 65 66a 68a Server 68c VM 63a RSVM Manager Server 68d Network 104 64 61 67 ➤ Gateway Router Server Manager 66b- A A 68b VM 63b 62b 60b FIG. 6 RSVM Manager

p.3Diagram illustrating a network setup with users submitting queries through the internet to a web site and the components of the web site.

-34 34 INTERNET HTML 32- WEB SERVER -34 41. 36(2) 36 36(1) DAILY QUERY LOG FIG. 1 48 30- WEB SITE 40- 38 QUERY SERVER SPELL CORRECTION PROCESS BIBLIOGRAPHIC DATABASE 36(M) TABLE GENERATION PROCESS CORRELATION TABLE 46 50

p.4Screenshot of a book search page with fields for Author and Title, as well as subject searches.

File Edit View Go Favorite Help ☑ Bock Forw... Stop Refresh Home Search Favorite Print Address http://www.amazon.com/book_search amazon.com Book Search Enter Author and/or Title Mail 42 Author: Exact Name Last, First Name Start of Last Name 43 Title: Exact Title Title Word(s) Start(s) of Title Words Search Now Clear the Form Author Search Tips / Title Search Tips Search by Subject Subject: 44 Exact Subject Start of Subject Subject Word(s) Start(s) of Subject Word(s) Search Now Clear the Form Subject Search Tips Other Search Methods: ISBN, Publisher/Date, Quick Search Amazon.com Home Music Search Your Account FIG.2

p.5Table of keywords and related search terms along with frequency counts.

60 COSMOS 64 JAVA BIKE ASTRONOMY (210) PROGRAMMING (320) TRAIL (280) SAGAN (180) COFFEE REPAIR (240) (190) UNIVERSE (111) API (120) MOUNTAIN (85) N Terms 62 SPACE (110) LANGUAGE (118) SCHWINN (99) CARL (90) MANAGEMENT (60) MOAB (19) ... ... FIG.3

p.6Flowchart illustrating the process of searching a query and attempting to correct a search query if no items are found.

PROCESS SEARCH QUERY RECEIVED FROM USER 70 ATTEMPT SEARCH 72 NO YES ITEMS FOUND=0 ? 74 RETURN LIST OF ITEMS YES -80 RETRIEVE RELATED TERMS LIST FOR EACH MATCHING TERM, AND MERGE IF MULTIPLE LISTS FOR EACH NON-MATCHING TERM: 84 76 DOES QUERY INCLUDE BOTH MATCHING AND NON-MATCHING TERMS ? NO RETURN NULL QUERY RESULT MESSAGE FOR EACH FIELD-CORRESPONDING RELATED TERM IN LIST, CALL SPELLING COMPARISON FUNCTION TO COMPARE NON-MATCHING TERM TO RELATED TERM DELETE NON-MATCHING TERM FIG.4 NO -90 86 TERM WITH SIMILAR SPELLING FOUND ? YES REPLACE NON-MATCHING TERM NEXT NON-MATCHING TERM PERFORM SEARCH WITH MODIFIED QUERY AND RETURN RESULT 88 94 78

p.7Text showing entries in the query log file.

100 102 Friday, 13-Feb-98 02:23:52 User Identifier = 29384719287 HTTP_REFERRER= http://www.amazon.com/book_search_page PATH_INFO=/book_search author = Seagal title Human Dynamics items_found = 2 : Friday, 13-Feb-98 02:24:11 User Identifier = 29384719287 HTTP_REFERRER= http://www.amazon.com/book_search PATH_INFO=/ISBN = 1883823064 : Friday, 13-Feb-98 06:15:03 User Identifier = 54730543261 HTTP_REFERRER= http://www.amazon.com/book_search_page PATH_INFO=/book_search subject biking China items_found = 0 Friday, 13-Feb-98 10:07:34 User Identifier = 027385918272 HTTP_REFERRER= http://www.amazon.com/music_search_page PATH_INFO=/music_search artist Miles Davis items_found = 22 : FIG.5

p.8Flowchart describing the process for generating the correlation table from query logs.

BEGIN 110 PARSE DAILY LOG FILE TO EXTRACT QUERY SUBMISSIONS FOR WHICH ITEMS FOUND > 0 112 CORRELATE TERMS BASED ON FREQUENCY OF OCCURRENCE WITHIN SAME QUERY 114 CREATE DAILY RESULTS FILE 116 MERGE DAILY RESULTS FILES FOR LAST M DAYS OVERWRITE EXISTING CORRELATION TABLE WITH NEW CORRELATION TABLE FIG.6 END 118

p.3This diagram shows a system configured to generate natural language responses using response templates.

101 111 Network 104 120 103 100 Skill(s) 190 Knowledge Graph 1 102 ASR 150 NLU 160 Q&A 161 Response Template(s) 180a Text data 105 Interface 122 食 112 FIG. 1 Response Template(s) 180N Knowledge Graph N

p.4This diagram illustrates the response template interface.

Attributes 222 216 Full Name Start Date Phone Number Job Title Bob Smith 9/15/2015 (123) 555-0100 Executive Assistant Entities 220 Cary Bobble 4/11/2018 (123) 555-0199 Founder and CEO Response Template Interface 122 111 User Interface 122 Response Template Name 204 When did employee start Natural Language Input 206 When did (People_Directory.Full_Name} (start|join) (company/org/team/organization() Response Type 208 Attribute Skill hand-off : : People Directory Start Date Tenure ◎ Disable Template Enable Template Q 210 212- FIG. 2 112 214- 218

p.5This diagram shows a data structure that associates response templates with account identifiers.

300- 103 Device ID Account ID Knowledge Graph(s) Location Encryption Response Algorithm Templates FST 302 1 0x0011111F A 308 314 Q&A 161 304 2 0x01111110 B 310 316 306 3 0x10100111 C 312 318 FIG. 3

p.6This diagram is a performance dashboard.

User Interface 422 111 Performance Dashboard 402 Natural language input 404 ◎ When did (People Directory. Full Name} join 410 412 112 414- : Suggested Response Template Auto-generated Response Template Manually Generate Response Template FIG. 4 Statistics 406 17%

p.7This diagram illustrates a computing device architecture.

Processing Element 504 500 Storage Element 502 Display Component 506 Input Device 508 Response templates 180 Power Supply 514 Transfer App 524 Operating System 522 Sensor(s) 530 Communication I/F SR I/F 534 512 Wireless 536 GPS 538 Mobile I/F 540 Wired 542 FIG. 5

p.8This diagram illustrates a system for sending and providing data.

62a 60a Network 104 64 61 67 Gateway Router 62b 60b FIG. 6 Data Center 65 66a 68a Server 68c VM A RSVM 63a Manager A Server Manager 66b- 68b Server 68d VM RSVM 63b Manager

p.11This diagram illustrates a speech processing system.

900 R&A 940 943 954 955 956 942 Context System 941 942 config rules policy Shortlister 910 Skill Proposal 914 Skill Query 916 Ranking component 920 Decider 932 904 Top K Skills 908 906 915 915,917 918 947 942 Feedback 997 Skill(s) 190 918 915 917 934 Dialog Speechlet 952 ASR NLU/Q&A 150 160/161 180 102 904 904 906,908 908 Routing service 912 Orchestrator 930 FIG. 9 TTS 936 934

p.1This diagram illustrates a cache-based skill selection for natural language processing.

102 ASR 150 NLU Component 160 Skill Selector 108 ASR output data 120 Auxiliary NLU process 114 108 1 Skill Selector 108 First NLU process 104 Skill Selector 108 | T Run NLU models on shortlisted skills 110 Merge and re-rank Q&A Process 103 112 Skill shortlister 142 Run NLU models on shortlisted skilis 144 Merge results 146 NLU output data 152

p.2This figure shows the components of a remote system usable for Natural Language Processing (NLP).

Wakeword detection component 221 Audio data 211 Audio 11 220 111 ASR 150 Skill selector 108 Orchestrator 230 Skills 290 Natural Language 160 FIG. 2 270 Text-to-Speech (TTS) 280

p.3This diagram shows the determination of semantically similar utterances for cache-based skill selection.

ASR Output Data 120 Embedding space 302 "Could you show me a low sodium Alfredo sauce please" 360b X Embedding 320a (ASR Output data 120) 360a Top-k closest utterances 360 360a "Order alfredo sauce" 362a General, 364a Ordering 362b General, 364b 360c "How do I make alfredo sauce" Food, X Video h 360c 360d : : 3600X FIG. 3 Union of skills associated with top-k closest utterances 360 New Cache entry 380 382 "Could you show me a low sodium Alfredo sauce please" General, Ordering, Food, Video 384

p.4This figure is a block diagram of an example architecture of a speech processing-enabled device.

Processing Element 404 Display Component 406 Input Device 408 Power Supply 414 400 Storage Element 402 Operating System 422 Cache-based Skill Selector 108 Transfer App 424 Voice Recognition Component 480 Sensor 430 Image Sen. 432 Microphone(s) 470 SRI/F 434 Communication I/F 412 Wired 442 GPS Interface 438 Wireless 436 Mobile I/F 440 FIG. 4

p.5This block diagram depicts the components of a remote computing device in a Natural language processing system.

590 I/O Device Interfaces 592 Processing Element(s) 594 Network 504 Memory 596 Cache-based Skill Selector 108 FIG. 5

p.6This is a flow chart illustrating cache-based skill selection for natural language processing.

610 612 614 616 618 600 Receiving first natural language input data Generating first automatic speech recognition (ASR) data representing the first natural language input data Searching a cache using the first ASR data Determining a first skill associated with the first ASR data in the cache Determining first intent data representing a semantic interpretation of the first natural language input data using a first natural language process associated with the first skill FIG. 6

p.7This conceptual diagram illustrates the natural language understanding processing.

Natural Language 160 160 Recognizer 763 NER 762 IC 764 FIG. 7 Knowledge Base(s) 772 776b 776n 774b Skill 2 Skill 2 Skill 1 Grammar Skill 1 Intents 776a 774a 43429 47429 Gazetteer B Gazetteer A 786n 786b- Skill 2 Skill 1 Lexicon 784n 784b 786a 784a

p.8This diagram illustrates a speech processing-enabled device and a speech processing management system.

Natural Language Processing-Enabled Device 111 Voice Services Component 826 Speech Comm Libr 836 Hybrid Request Selector 832 Natural language Processing System 220 Hybrid Hybrid Exec LRO Proxy Cntlr 842 Natural Language Processing Component 140 834 838 Local Natural language Processing Component 140 Local ASR Component 150 Local NLU Component 160' ASR Component 150 NLU Component 160 Text-to-Speech Component 280 Skills 290 Skill Selector 108 Speech Interaction Manager 828 Local TTS Synthesis Component 280 App 830 Audio Front End 825 Skills Execution Component 844 Wakeword Engine 824 Output Device 810 Orchestrator 230 Microphone(s) 162 Communications Interface 812 FIG. 8 Audio Data 102 Storage 270

p.9This diagram illustrates cache-based intent determination and cache-based slot data determination.

102 ASR 150 NLU Component 160 First NLU process 104 Utterance 902 Utterance 902 Skill Selector 108 Skill Selector 108 NER 762 Slot data 908 String data 922 To routing (906,908) component Cache 904 Intent data 906 {Intent data 906, Slot data 908) IC 764 Cache building 910 Runtime processing using populated cache 920 FIG. 9

p.10This diagram depicts an example storage component for caching and retrieval of data.

1000 String data 1002 Play Happiness What is the weather tomorrow Show me how to make tomato sauce Play Moonshot : Skill identifier(s) 1004 Intent data 1006 Play_movie, Play_song, Play_artist Slot data 1008 Happiness Weather Output_forecast tomorrow Food, Video Play_video Tomato sauce Movie, Music FIG. 10

p.4A block diagram illustrating a process for recognizing text using multiple engines and confidencing modules.

Image, Engine 1, Engine 2, Engine N, Confidencing Module 1, Confidencing Module 2, Confidencing Module N, Combination

p.4An image showing a user pointing a phone at a sign and the text that is trying to be extracted.

The Beach Hut (310) 123-4567

p.5A visual representation of a process where an image of text is analyzed by multiple recognition engines, each producing a slightly different output with a confidence score.

The Beach Hut (310) 123-4567, Engine 1, Engine 2, Engine N, The Beaeh hat (310)128-9567, The Beach Hut (310)123-4561, The beach hut (310)123-4567, Confidencing Module 1, Confidencing Module 2, Confidencing Module N, 80%, 70%, 85%, Combination

p.6A diagram of an example system using mobile devices, a network, and an optical character recognition system.

NETWORK, Optical Character Recognition System, Image Processing Module, Optical Character Recognition Module

p.7A flowchart illustrating a process for recognizing text by obtaining an image, analyzing it, selecting a region of text, and processing it simultaneously with multiple recognition engines.

Obtain an image, Analyze image to locate text, Select region of located text, Binarize selected region of text, Simultaneously process region with at least two recognition engines, Compare processed results, Determine consensus text string

p.8Diagrams showing computing devices from different angles.

コ

p.8A block diagram of the components of an electronic computing device.

Memory, Processor, Display, Imaging Element, Orientation Determining Element, Positioning Element

p.9A diagram showing a network with client devices and servers.

NETWORK, WEB SERVER, APPLICATION SERVER, CONTENT, SESSION, USER INFORMATION

p.2This is a view of a geographic region on a computing device with POIs displayed within and outside of a bounding box.

104 106 108 112 102 114 FIG. 1A 100 116

p.3This table summarizes query details and results, including query keywords, viewport coordinates, user location, POI title, and address.

Query ld Viewport User Location Results Query Details Foo Store 12345 (latitude x, longitude y), (latitude q, longitude u) (latitude t, longitude w) סו Rank Title Address 701 1 Foo Store Foo Address 1 702 2 Foo Store Foo Address 2 703 3 Foo Gallery Foo Gallery Address 1 704 4 Foo Outlet Store Outlet Address 1 705 5 Foo Distribution Center Distribution Center Address 1 706 6 Depot for Foo Goods and Items Depot Address 1 ...

p.4This graph illustrates a reverse logistic function where the x-axis represents distance in kilometers, and the y-axis represents a distance score.

1.5 1+0.0033 x FIG. 2 206 204

p.5This is a view of a geographic region on a computing device with coffee-related POIs displayed in a map.

ABC Coffee 306 Lindoin Ave. XYZ Movie Theater 300 -304 302 XYZ Book Store 308 Joe's Coffe ZAX Coffee Clinton Rd. 123 Coffee 310 312

p.6This graph represents a logarithmic distance function with the x-axis representing distance, and the y-axis representing the logarithmic distance value.

1.5 1+ln(1+x) FIG. 4

p.1Shows examples of product listings, with a first listing for 'dresses' and a second for 'formal dresses'.

dresses Party Dress by Acme $74.99 Block Dress by Closet Pop $52.50 Short Stripes by HappyClothes $87.99 formal dresses Party Dress by Acme $74.99 Block Dress by Closet Pop $52.50 Block Dress by Closet Pop $52.50

p.2Illustrates example representations of parts-based visual similarity search using sleeve style, neckline, cut, hemline, and color.

Sleeve Style Neckline Cut Hemline Color Sleeve Style Neckline Cut Hemline Color Sleeve Style Neckline Cut Hemline Color

p.3Illustrates an example technique for training a database for parts-based visual similarity search using neckline, sleeves, and hemline attribute databases.

Neckline Attribute Database Sleeves Attribute Database Hemline Attribute Database

p.4Illustrates an example technique for parts-based visual similarity search using rankings by attribute.

Sleeve Length Neckline Cut Hemline Color Sleeves Attribute Database Hemline Attribute Database Weighting Function Aggregated Ranking

p.5Depicts an example environment in which aspects of the various embodiments can be implemented including a third party provider.

Network Third Party Provider Content server Recommend. Engine Visual Attribute Component Data Training Component Weighting Component

p.6Shows an example process for training a database for parts-based visual similarity search.

Obtain image data representing an item in an electronic catalog of items Determine portions of the image data corresponding to visual attributes Extract feature vectors of the portions of the image data Associate the feature vectors with the item and the corresponding visual attributes

p.7Displays an example process for parts-based visual similarity search.

Receive a query associated with a first visual attributes and a second visual attribute Determine a first query feature vector associated with the first visual attribute Determine first feature vectors for a plurality of items Determine first attribute similarity scores for the plurality of items Determine a second query feature vector associated with the second visual attribute Determine second feature vectors for the plurality of items Determine second attribute similarity scores for the plurality of items

p.8Describes an example process of updating search results based on parts-based search refinement.

Receive a query associated with one or more visual attributes of one or more items Determine one or more query feature values of the one or more visual attributes Query a database containing items with feature values corresponding to the visual attributes Compare the query feature values to the corresponding feature values of the items for the one or more visual attributes Determine attribute similarity scores for the items for individual visual attributes Determine overall similarity scores for the plurality of items based on the attribute similarity scores Select items to display based on overall similarity scores Receive new query attributes? Update the one or more visual attributes

p.9Depicts an example computing device that can be used in accordance with various embodiments.

Memory Communication Component Processor Input Device Display

p.10Shows an example environment for implementing aspects in accordance with various embodiments.

Network Web Server Application Server Content Session User Information

p.3The system diagram illustrates the components of the search engine system, including web crawlers, product score generators, index tools, query servers, and databases.

INTERNET WEB CRAWLER PRODUCT SCORE GENERATOR INDEX TOOL HTML WEB SERVER QUERY LOG QUERY SERVER PRODUCT SPIDER/DATABASE KEYWORD URL TITLE SQUIB SCORE CATEGORY SPELL CHECKER RANKING PROCESS SEARCH TOOL COSMOS www.abc.com www.def.com www.geh.com COSSACK www.mno.com www.pqr.com www.stu.com : : BOOKS MUSIC AUCTIONS VIDEO SOFTWARE ELECTRONICS

p.4This figure shows a sample search tool interface page for a web site.

File Edit View Go Favorite Help Address WELCOME TO amazon.com SEARCH BROWSE BOOKS MUSIC VIDEO AUCTIONS ALL PRODUCTS BOOKS Bestsellers, Computers Kids, Business MUSIC Top Sellers, New Releases Soundtracks VIDEO DVD's, Top Sellers New Releases AUCTIONS How Auctions Work Collectible, Sports Mark Twain

p.5This figure represents a sample results page for an 'All Products' search within a commercial website.

Address Top search results from Amazon.com for Mark Twain BOOKS Letters from the Earth Following the Equator: A Journey Around the World Joan of Arc See all matching results in Books VIDEOS Mark Twain Tonight (1967); VHS The Adventures of Mark Twain (1944); VHS Mark Twain (1995) A&E Biography, VHS See all matching results in Videos... AUCTIONS 10 Takes from Mark Twain on CDROM current bid: $9.99 Mark Twain and the Laughing River, Audio CD current bid: $7.99 Mark Twain: The Musical (1991) current bid: $11.99 See all matching results in Auctions... MUSIC See all matching results in Music... Additional Matches for Mark Twain from other on-line merchants: SOFTWARE A Horse's Tale Extracts from Adam's Diary A Visit to Heaven See all matching results in Software... ELECTRONICS See all matching results in Music... RELATED PRODUCTS For Mark Twain on the Web

p.6This figure depicts an example of a results page showcasing 'Related Products' items from external online retailers.

Address Additional matches from the Web 1418 matches for Mark Twain from other merchants on the Web. Too many products? Refine your search. Items 1-5 Mark Twain: Wild Humorist of the West Browse or Search our Catalog View Shopping Cart Sound Samples... Vintage Lifestyles - A visit to Mark Twain's House A visit to Mark Twain's house by Garrison Keiller... Celebrated Jumping Frog of Calavaras County Check out our Award-winning Titles... Mark Twain Project T-shirts Unique Mark Twain T-shirts! American Original! Autobiography of Mark Twain Edited by Charles Neider, read by Michael Anthony EXIT Rate This Merchant

p.7This flow chart describes the process for generating the product spider database.

WEB CRAWL A PORTION X OF THE WORLD WIDE WEB APPLY PAGE ANALYZER SUBMIT URL'S FROM PREVIOUS PRODUCT SPIDER DATABASE FOR RECHECKING PAGE SATISFY FILTER DISCARD YES SUBMIT URL TO WEB CRAWLER FOR RECHECKING WEB CRAWL SUBMITTED PAGES APPLY PAGE ANALYZER PAGE SATISFY FILTER DISCARD YES APPLY INDEX TOOL

p.8This diagram shows the workflow for returning the All Products search results.

PROMPT FOR ALL PRODUCTS QUERY RECEIVE SEARCH QUERY APPLY THE QUERY TO ALL CATEGORIES RETURN QUERY RESULTS FROM EACH CATEGORY TO QUERY SERVER DETERMINE A RELEVANCE RANKING FOR EACH COMPETING CATEGORY ARRANGE COMPETING CATEGORIES FOR DISPLAY GENERATE SEARCH RESULTS PAGE

p.9This illustrates the structure of a database including a full-text index and popularity score table.

BOOKS FULL TEXT INDEX KEYWORD ITEM IDENTIFIER TWAIN TWANG TWEAK ITEM IDENTIFIER POPULARITY SCORE BOOKS POPULARITY SCORE TABLE KEYWORD MARK TWAIN

p.10This flowchart depicts the process for relevance ranking generation in the categories.

BEGIN FOR EACH SET OF COMPETING CATEGORIES FOR EACH CATEGORY DETERMINE A CATEGORY POPULARITY SCORE FROM CONSTITUENT SEARCH RESULT ITEMS NEXT CATEGORY CREATE A CATEGORY RANKING FROM THE CATEGORY POPULARITY SCORES NEXT SET END

p.11This figure outlines the search operation for all products.

SEARCH AUCTIONS DATABASE SEARCH POPULARITY SCORE TABLES YES GO TO BOX 650 IN FIG. 6 NO SEARCH FULL TEXT INDEXES YES NO RUN SPELL CHECK REPEAT BOXES 910-935 YES NO REPEAT BOXES 910-935 WITH SINGLE TERM QUERIES YES NO GENERATE NO RESULTS PAGE

p.3This diagram shows the basic components and process flow of a system that embodies the invention.

DATABASE ACCESS SYSTEM QUERY SERVER DATABASE STRING EXTRACTION AUTOCOMP SERVER SEARCHES SEARCHES BROWSER AUTOCOMP CLIENT BROWSER AUTOCOMP CLIENT DATASET OF AUTOCOMP STRINGS FIG. 1

p.4This image illustrates an example user interface suggesting autocompletion strings.

amazon.com Search: SO_ SONY SOFTWARE SONGS SOCKS GO! SONY VCR SONGS FROM THE ATTIC SO LONG AND THANKS FOR ALL THE FISH FIG. 2A

p.4This image illustrates an example user interface suggesting autocompletion strings.

amazon.com Search: SONY_ SONY VCR SONY TV SONY TRINITRON SONY HANDYCAM GO! FIG. 2B

p.5This diagram shows a trie data structure for storing autocompletion strings.

ROOT A B S Z ZA AA AZ OS NOS SONGS (8)... SONY (9) SONY HANDYCAM (8) SONY TV (8.5) SONY VAIO (5) FIG. 3

p.6This image shows a merchant web site setup, the various server components, and data flow.

MERCHANT WEB SITE PRODUCTS DATABASE(S) CUSTOMER DATABASE QUERY SERVER WEB SERVER AUTOCOMP SERVER QUERY LOG FIG. 4 DATA EXTRACTION

p.7This flow chart shows a method for extracting autocompletion strings from a query log.

GENERATE DATASET FROM QUERY LOG GET QUERY LOG DATA FOR LAST M DAYS FILTER OUT QUERIES THAT PRODUCED NULL QUERY RESULT OR REQUIRED SPELL CORRECTION OPTIONALLY PARTITION QUERY DATA ACCORDING TO USER GROUPS, AND PERFORM REMAINING STEPS FOR EACH USER GROUP IDENTIFY AND ASSIGN SCORES TO MOST FREQUENTLY USED SEARCH TERMS AND PHRASES STORE RESULTS IN TRIE OR OTHER DATA STRUCTURE LAST GROUP ? YES END FIG. 5

p.8This flow chart shows a method for generating dataset from item descriptions.

GENERATE DATASET FROM ITEM DESCRIPTIONS GET USER PURCHASE HISTORIES FOR LAST X DAYS OPTIONALLY PARTITION PURCHASE HISTORY DATA ACCORDING TO USER GROUPS, AND PERFORM REMAINING STEPS FOR EACH SUCH GROUP IDENTIFY Y BEST SELLING ITEMS FOR EACH BEST SELLING ITEM, EXTRACT CHARACTERIZING TERMS AND/OR PHRASES AND ASSIGN SCORES STORE RESULTS IN TRIE OR OTHER DATA STRUCTURE LAST GROUP ? YES END FIG. 6

p.3Example of a window of display content that can be presented in accordance with various embodiments.

Web Browser FILE EDIT VIEW FAVORITES TOOLS HELP Address Search: Acme Model One GO Submit + Features Brand Acme (17) Brand (13) CompU (23) TechCo (2) PEAR (31) + Price Category Smartphones (17) Smartphone Accessories (13)

p.4Example of system architecture for performing the disclosed embodiments.

Users Network Content Server External Content Providers Search Data

p.5Another example system architecture for performing the disclosed embodiments.

Client Devices External Content Browser / Application Content Console Network Interface / Networking Display Content Query Receiver Generator Crawler/ Webpage Analyzer Content Database Query Reviewer Hyperlink Retriever & Indexer Search Database

p.6Exemplary categorical hierarchy of items potentially related to a search query.

Acme Acme Model Apparel Shirts Toys Drones Cellular Telephones Smartphones Smartphone Accessories

p.7Another example of a window of display content that can be presented in accordance with various embodiments.

Web Browser FILE EDIT VIEW FAVORITES TOOLS HELP Address Search: Acme Model One GO Submit + Features Brand Acme Model One - "One Mile Radius" - 15 Channels Extended Battery Life Acme (17) Brand (13) CompU (23) Brand One Model - 5"SmartPhone Secure Model - GSM/CDMA TechCo (2) Brand Two Smartphone Cover PEAR (31) + Price Category Smartphones (17) Smartphone Accessories (13) Toys (5) - For 6" Smart Phone - Transparent Elastic holder Acme Model Shirt - Cotton - White with stripes - Moisture Wicking

p.8Distributed representation vectors

DISTRIBUTED REPRESENTATION VECTORS ACME MODEL ONE NEW DRONE 0.73 0.80 DRONE 0.99 0.98 DEVICE → 0.23 0.34 SMARTPHONE 0.07 0.08 RELEASED 0.88 0.90 NEWLY 0.95 0.96 ELECTRONICS 0.43 * ACCESSORIES 0.48 TOY 0.85

p.8Semantic processing for machine learning

RELEASED DRONE NEW Y-Axis (ACME) SMARTPHONE X-Axis (MODEL ONE)

p.9Examples of semantic processing for machine learning to determine common textual features found in external content.

External Website DATE MONTH DD, YYYY ACME DRONE ONE HAS HIT THE MARKET ACME'S NEW DRONE MODEL, THE ACME DRONE ONE RELEASED TODAY AND IS AVAILABLE AT MAJOR RETAILERS. THE MODEL 1 NEWLY RELEASED DRONE HAS IMPRESSIVE SPECIFICATIONS PROVIDED IN <HYPERLINKED WEBSITE>, INCLUDING A ONE MILE RADIUS....

p.10Example process flow to configure computing system of at least one processor for enhancing search query results with external content.

Using web crawler, seller accesses content external to the seller's online marketplace Determine that an identifier in the external content relates to at least one entry in the seller's search engine data store Extract at least one instance of data or metadata appearing in the external content Crawler found additional identifiers? Index the search engine data store to reflect association(s) between the extracted instance(s) of data and the pertinent entry(ies) in the data store When responsive to customer query, generate display content based on the association(s) and provide the display content to customer device

p.11Example components of a computing device.

Communication Component Memory Imaging Element Power Component Processor Audio Element Input Element Display Positioning Element

p.2A diagram showing the system architecture for the search engine with spell correction.

INTERNET WEB SERVER DAILY QUERY LOG WEB SITE QUERY SERVER SPELL CORRECTION PROCESS BIBLIOGRAPHIC DATABASE TABLE GENERATION PROCESS CORRELATION TABLE

p.3A screenshot of Amazon's book search page, showing input fields for author, title, and subject.

amazon.com Book Search Enter Author and/or Title Author: Title: Search by Subject Subject:

p.4A table illustrating the general format of the correlation table, containing keywords and their related terms along with correlation scores.

JAVA BIKE COSMOS COSMOS ASTRONOMY SAGAN UNIVERSE SPACE CARL PROGRAMMING COFFEE API LANGUAGE MANAGEMENT TRAIL REPAIR MOUNTAIN SCHWINN MOAB N Terms

p.5A flowchart detailing the steps performed by the query server in processing a query submission, including attempt search, query include matching and non-matching terms, delete non-matching term, perform search with modified query and return result.

PROCESS SEARCH QUERY RECEIVED FROM USER ATTEMPT SEARCH ITEMS FOUND=0 RETURN LIST OF ITEMS DOES QUERY INCLUDE BOTH MATCHING AND NON-MATCHING TERMS RETURN NULL QUERY RESULT MESSAGE FOR EACH NON-MATCHING TERM: FOR EACH FIELD-CORRESPONDING RELATED TERM IN LIST, CALL SPELLING COMPARISON FUNCTION TO COMPARE NON-MATCHING TERM TO RELATED TERM TERM WITH SIMILAR SPELLING FOUND DELETE NON-MATCHING TERM REPLACE NON-MATCHING TERM NEXT NON-MATCHING TERM PERFORM SEARCH WITH MODIFIED QUERY AND RETURN RESULT

p.6Examples from the query log, illustrating data collected from user searches and their interactions with the site.

Friday, 13-Feb-98 02:23:52 User Identifier = 29384719287 HTTP_REFERRER= http://www.amazon.com/book_search_page PATH_INFO=/book_search author = Seagal title = Human Dynamics items_found = 2 Friday, 13-Feb-98 02:24:11 User Identifier = 29384719287 HTTP_REFERRER= http://www.amazon.com/book_search PATH_INFO=/ISBN = 1883823064 Friday, 13-Feb-98 06:15:03 User Identifier = 54730543261 HTTP_REFERRER= http://www.amazon.com/book_search_page PATH_INFO=/book_search subject = biking China items_found = 0 Friday, 13-Feb-98 10:07:34 User Identifier = 027385918272 HTTP_REFERRER= http://www.amazon.com/music_search_page PATH_INFO=/music_search artist = Miles Davis items_found = 22

p.7A flowchart illustrating the process of generating the correlation table from query logs.

PARSE DAILY LOG FILE TO EXTRACT QUERY SUBMISSIONS FOR WHICH ITEMS FOUND > 0 CORRELATE TERMS BASED ON FREQUENCY OF OCCURRENCE WITHIN SAME QUERY CREATE DAILY RESULTS FILE MERGE DAILY RESULTS FILES FOR LAST M DAYS OVERWRITE EXISTING CORRELATION TABLE WITH NEW CORRELATION TABLE BEGIN

p.3This is a network diagram of an example system architecture for generating modified search results based on query fingerprints.

USER DEVICE 1 102 Q1 122 MODIFY USER EXPERIENCE BASED ON FINGERPRINT INFORMATION 118 USER DEVICE 2 104 USER DEVICE 3 106 Q2 Q3 NETWORK 110 ELECTRONIC MARKETPLACE 112 GENERATE FINGERPRINT INFORMATION FOR QUERY USER DEVICE N 108 USER BEHAVIOR DATABASE 114 QUERY CLASSIFICATION DATABASE 116 FIG. 1 120 100 DETERMINE QUERY CLASSIFICATION BASED ON FINGERPRINT INFORMATION

p.4This figure is a system architecture illustrating the components involved in generating modified search results based on query fingerprints.

PROCESSOR(S) 214 MEMORY 212 BROWSER APPLICATION 206 SERVICE PROVIDER 210 204(1) 204(N) USER DEVICE(S) USER(S) 202 NETWORK(S) 208 O/S 226 DATA STORES 228 MEMORY 216 QUERY FINGERPRINT GENERATION AND CLASSIFICATION PROCESSOR(S) 218 STORAGE 220 COMM. CONN. 222 SERVICE 230 I/O DEVICE(S) 224 FIG. 2

p.5An architecture for query fingerprint generation and classification service is depicted in this diagram.

QUERY FINGERPRINT GENERATION AND CLASSIFICATION SERVICE 230 APPLICATION PROGRAMMING INTERFACE 312 GRAPHICAL USER INTERFACE 314 QUERY FINGERPRINT GENERATION MODULE 318 ELECTRONIC CATALOG 316 QUERY FINGERPRINT CLASSIFICATION MODULE 320 COMMUNICATIONS MANAGER 324 DYNAMIC NETWORK CONTENT GENERATION MODULE 322 FIG. 3 USER BEHAVIOR DATABASE 114 QUERY CLASSIFICATION DATABASE 116

p.6This flowchart presents a process for generating a query fingerprint for queries in an electronic marketplace.

IDENTIFY A QUERY SUBMITTED BY ONE OR MORE USERS DURING A SEARCH SESSION 402 IDENTIFY A FIRST ACTION PERFORMED BY THE USERS IN RELATION TO THE QUERY 404 STORE THE FIRST ACTION AND CORRESPONDING TIME INFORMATION AS FINGERPRINT INFORMATION FOR THE QUERY 406 ANALYZE ACTIONS PERFORMED BY USERS AFTER IDENTIFIED ACTION 408 IDENTIFY A SUBSEQUENT ACTION BASED ON THE ANALYSIS 410 STORE THE SUBSEQUENT ACTION AND CORRESPONDING TIME INFORMATION BASED ON THE ANALYSIS 412 YES 414 NO SUBSEQUENT ACTION EXIT ACTION? GENERATE FINGERPRINT FOR QUERY 416 FIG. 4

p.7This is a flowchart for classifying queries based on fingerprint information.

RECEIVE A QUERY ISSUED BY A USER 502 ACCESS A QUERY CLASSIFICATION DATABASE HAVING FINGERPRINT INFORMATION 504 DETERMINE A FINGERPRINT FOR THE QUERY BY COMPARING THE FINGERPRINT INFORMATION TO AN ACTION PERFORMED BY THE USER IN RELATION TO THE QUERY 506 DETERMINE CLASSIFICATION FOR QUERY BASED ON THE FINGERPRINT 508 FIG. 5

p.8The flowchart depicts a process for modifying user experiences for a query based on fingerprint information.

RECEIVE A QUERY FROM A USER 602 IDENTIFYING A FIRST ACTION PERFORMED BY THE USER IN RELATION TO THE QUERY 604 ACCESS A CLASSIFICATION DATABASE HAVING FINGERPRINT INFORMATION 606 FINGERPRINT NO INFORMATION AVAILABLE? 606 YES ACCESS FINGERPRINT INFORMATION FROM CLASSIFICATION DATABASE TO IDENTIFY A QUERY CLASSIFIED UNDER SAME CATEGORY AS USER QUERY 608 DETERMINE FINGERPRINT FOR QUERY 610 DETERMINING A CLASSIFICATION FOR QUERY BASED ON FINGERPRINT 612 MODIFY USER EXPERIENCE FOR USER BASED ON FINGERPRINT AND CLASSIFICATION 614 FIG. 6

p.12This network diagram illustrates the components involved in dynamic network content generation.

DYNAMIC NETWORK CONTENT GENERATION MODULE 322 LAYOUT GENERATION MODULE 1004 CATEGORY IDENTIFICATION MODULE 1008 INTERACTIVE NETWORK REGION GENERATION MODULE 1006 1002 RENDERING ENGINE 1010 FIG. 10

p.1This is a screenshot of a search interface showing suggested queries based on user input.

Please Enter Your Search Query: Bo GO Boston, Massachusetts Boston Electronics Book stores near Palo Alto Bob's Burgers The Boston Redsox Bowl Game Schedule

p.3This is a screenshot of a search interface showing search results for 'Boston Electronics' based on the selected query and hint.

Boston Electronics GO Current Category: Electronics X 1. Boston Electronics - Your Place to Buy Electronics!! Boston Electronics offers a wide arrange of electronic devices for professionals, schools, and your every day garage inventor. 2. Electronic Warehouse - Wholesale Electronic Devices Electronic Warehouse offers the quantity you need, at the price you deserve. 3. Digital Superstore - Appliances, Computers, and Stereos A one-stop shop for all of your needs. Computers, Stereos, CDs, DVDs, Smart Phones, Refrigerators, Washing Machines, Dryers, and more! 4. Dale's Electronics Repair Visit Dale's Electronics Repair offers computer repair and more. Dale's also sells a wide variety of electronic devices. 5. Chips For Cheap - Processors, Memory, and more! Chips For Cheap sells the chips your company needs at an affordable price.

p.4This diagram illustrates the system architecture for query processing and hint management.

Query + Hint Network Suggested Query Creation Module Search System Machine Learning Engine Search Result Provider Module Data Store

p.5This flowchart represents the process of sending a search query and getting search results based on hints.

Send a search query to a search system Receive one or more suggested queries for the search query Present the suggested queries Either suggested query selected, or entered search query is identical to a suggested query? Receive a suggested query Send the suggested query and at least one respective hint associated with the suggested query to the search system Receive a listing of the search results that are responsive to the suggested query and the respective hint from the search system Send the search query to the search system Receive a listing of the search results that are responsive to the search query from the search system Present the listing of search results on the display of the computing device

p.6This is a block diagram showing components of a computing device.

Memory Processor Display Image Capture Element Motion Component Audio Component

p.7This is a network diagram illustrating the connectivity of different devices, servers, and a data store.

Network Web Server Application Server Content Session User Information

p.5This is an example of an item rating table.

item rating table item term score identifier : dynamics 0801062272 1 dynamics 1883823064 22 dynamics 9676530409 7 human 0814403484 16 human 1883823064 45 human 6303702473 3 :

p.7This is a diagram showing generation of rating tables for composite periods of time from rating tables for constituent periods of time.

item rating tables constituent period constituent period item rating table composite period composite period item rating table 8-Feb-98 9-Feb-98 10-Feb-98 11-Feb-98 12-Feb-98 13-Feb-98 8-Feb-98 to 12-Feb-98 9-Feb-98 to 13-Feb-98

p.8This is an example of an item rating table for a composite period.

item rating table item term score identifier dynamics 0801062272 4 dynamics 1883823064 116 dynamics 1887650024 2 dynamics 9676530409 45 human 0814403484 77 human 1883823064 211 human 6303702473 12

p.1This diagram shows a user interface where a question is asked, an answer is generated, a noun phrase in the answer is highlighted, and automated product search results are displayed based on the generated noun phrase.

User Interface 100 Question 110 Input text bar 112 -116 How do you take off gel nails? Answer: Removing gel nails involves soaking them in pure acetone Answer 120 generated automatically Noun phrase 130 132 Brand A Brand B Automated product search results 140 based upon generated Noun phrase

p.3This diagram illustrates a user interface showing a question, an answer with highlighted noun phrases and n-grams, and automated product search results and a user review associated with the n-grams.

Is company A mouse comfortable? Question 210 User Interface 200 Input text bar 212 Noun phrase 1 N-gram 1 Company A's newest mouse is comfortable, customizable, and easy to use 250 ADD TO CART Answer 220 I love this mouse. Automated product search results 230 based upon generated N- gram User Review 240

p.4This diagram shows a system diagram for generating a UI with highlighted noun phrases and associated products, involving a user interface, network, answer generation model, semantic similarity model, product search engine, and product database.

User Interface 302 310 Question 314 User Interface Answer 316 Network 300 340 320 Click Training Data 342 Alternative Training Data Training 330 Question 314 Answer Generation Model Answer 326 Question 314 Semantic Similarity Model Noun Phrases 348 in Answer Ranked Noun phrases 350 UI Generator Product Images 370 Answer 316 with Noun Phrases Highlighted And Associated With Products 372 Product Search Engine 318 Product Database 360 362

p.5This diagram shows the detailed components of the semantic similarity model, including semantic similarity modules, scores, a controller, and training data.

Question 314 Training 440 Simantic Similarity Score 420 Answer 326 330 Noun Phrase ID 408 Semantic Similarity Model Noun Phrase 410 Question Answer 314 326 416 Answer 326 412 414 Simantic Similarity Simantic Similarity Training 440 Score 422 Training 440 Score 424 Controller Ranking of Noun Phrases 350 430 Click Training Data 340

p.6This diagram shows an example of converting a question-answer pair into a ranked output using the semantic similarity model, including noun phrase identification and scoring.

510 520 530 350 (1) A question-answer pair Q: How to make porcelain tile shine? A: Regular damp mopping with an all purpose cleaner or any floor cleaning agent, is all what you need to keep your porcelain flooring shining for years. (2) Pool of candidate n- grams Q: How to make porcelain tile shine? A: Regular damp mopping with an all purpose cleaner or any floor cleaning agent is all what you need to keep your porcelain flooring shining for years (3) Ranking candidates Query recommendation methods 540 floor cleaning agent Regular damp Porcelain flooring floor • Noun phrases Patent Application Publication Oct. 3, 2024 Sheet 5 of 10 US 2024/0331004 A1 sim(ngram, Q) 542 sim(ngram, A 544 sim(ngram, Q + A) 1546 Sim(ngramQ) + sim(ngram, A) 2 The top n are recommended as queries

p.7This diagram illustrates how click data is obtained for training the semantic similarity model by analyzing different searches leading to the same product.

Obtaining Click Training Data 610 613 Input text bar 612 Are car seat bases interchangeable? Generally, car seat bases can be interchanged. Answer 614 Seat type 1 Seat type 2 Product ID 1 Search 1 616 Input text bar 612 Universal infant car seat 630 Related searches due to finding same Product ID 1, which is clicked upon. 632 Seat type 1 Seat type 2 Product ID 1 640 Search 2

p.8This diagram shows a computing system within a network-based compute service provider, including server computers, hypervisors, instances, semantic similarity models, and other components.

CLUSTER OF SERVER COMPUTERS SERVER COMPUTER INSTANCE 706A 702A 708 HYPERVISOR COMPUTE SERVICE PROVIDER 700 730 SERVER COMPUTER 702B INSTANCE 706B HYPERVISOR LOCAL AREA NETWORK 708 702D 708 INSTANCE 706D 330 SERVER COMPUTER HYPERVISOR SEMANTIC SIMILARITY MODEL SEMANTIC SIMILARITY MODEL TRAINING DATA ACQUISITION 752- SERVER COMPUTER 702C 712 AUTO SCALING COMPONENT INSTANCE 706C 710 MANAGEMENT COMPONENT HYPERVISOR 708 714 DEPLOYMENT COMPONENT TO WIDE AREA 715 USER ACCOUNT NETWORK 740 704 SERVER COMPUTER

p.9This flowchart illustrates a process for recommending products in a UI, including receiving a question, extracting n-grams, ranking them, and generating the display.

810 RECEIVE A QUESTION FROM A CLIENT COMPUTER 820 RECEIVE AN ANSWER TO THE QUESTION EXTRACT A POOL OF CANDIDATE N-GRAMS INTO A SEMANTIC SIMILARITY MODEL RANK THE N-GRAMS USING THE SEMANTIC SIMILARITY MODEL GENERATE FOR DISPLAY THE PRODUCT IN ASSOCIATION WITH THE N-GRAMS 830 840 850

p.10This flowchart illustrates a process for recommending products, starting from a user question and ending with searching for and selecting products based on the ranking of n-grams.

910 RECEIVE A QUESTION FROM A USER INTERFACE 920 RECEIVE AN ASSOCIATED ANSWER TO THE QUESTION 930 EXTRACT N-GRAMS FROM THE GENERATED ANSWER 940 GENERATE SCORES FOR THE N-GRAMS 950 RANK THE N-GRAMS USING THE SCORES SEARCH FOR AND SELECTING PRODUCTS BASED UPON THE RANKING 960

p.11This diagram shows a generalized example of a suitable computing environment in which the described innovations may be implemented.

COMPUTING ENVIRONMENT 1000 1030 central processing unit 1010 graphics or co- processing unit 1015 MEMORY 1020 MEMORY 1025 COMMUNICATION CONNECTION(S) 1070 INPUT DEVICE(S) 1050 OUTPUT DEVICE(S) 1060 STORAGE 1040 SOFTWARE 1080 IMPLEMENTING DESCRIBED TECHNOLOGIES

p.2This is a table outlining an example of content item ranking using rank products, with different features (A, B, C), their ranks, the calculated rank product, and the resulting final order of the items.

Network 104 Original Order A B C A-rank B-rank C-rank Rank Product Final rank Final Order Item #1 3 2 6 3 4 1.5 18 3 Item #2 102 Item #2 6 7 6 1 1 1.5 1.5 1 Item #4 Item #3 2 4 4 4 2.5 4 40 4 Item #1 Item #4 5 4 5 2 2.5 3 15 2 Item #3 103

p.3This figure shows a pseudo-code implementation for the rank product algorithm.

Network 104 102 103 240 1. For each feature n of N Compute the rank of the i-th item, rx(t) when sorting from high to low values: rank 1 corresponds to the highest value, rank N to the lowest 2. Compute the product of the ranks (Le., rank product) at the i-th position: P(1) = (1) 3. Compute the rank R(1) of P(1) when sorting from low to high values: R(1) = rank (P(i)); rank 1 corresponds to the lowest value, rank N to the highest 4. Re-order and display the M items according to R(1)

p.4This table presents combinatorial feature selection results and the associated Mean Reciprocal Rank (MRR) gain.

Network 104 Combinatorial Feature selection 310 Feature Combination Baseline MRR RP MRR MRR Gain (%) qaa, pds, csf 0.3069 0.3098 +0.94 102 qaa, pds, csf, bm25 0.3069 0.3078 +0.29 gap, pds, csf 0.3069 0.3074 +0.15 qap, qaa, pds, csf, bm25, rc 0.3069 0.3069 +0.00 qaa, pds, csf, bm25, re 0.3069 0.3068 -0.04 103

p.5This is a table comparing total revenue and total sales for an e-commerce service, with rank product implementations versus a baseline ranker.

U.S. Patent Nov. 14, 2023 Sheet 4 of 7 US 11,816,720 B1 SETUS IVLOL VIOL TREATMENT ONLINE SESSIONS REVENUE RELATIVE CHANGE P-VALUE SALES RELATIVE CHANGE P-VALUE Baseline 3,208,901 $101,814,814 1,224,192 73 3,210,805 $101,730,114 -0.07% (-0.46%, 0.31%) 0.713 1,226,848 0.17% (0.01%, 0.35%) 0.069 0.03% 3,214,135 $101,923,002 0.897 1,229,189 (-0.36%, 0.41%) 0.27% (0.09%, 0.45%) 0.004 ٤٤ 3,212,978 $101,929,728 0.01% (-0.38%, 0.40%) 0.963 1,226,959 0.13% (-0.05%, 0.31%) 0.170

p.6This is a flowchart illustrating a process for content ranking using rank products.

400 410 420 Receiving a query 430 Generating, based on the query, a first ranked list of content items ranked based on a first optimization objective and/or based on a first feature Generating, based on the query, a second ranked list of content items ranked based on a second optimization objective and/or based on a second feature 440 Determining, for a first content item, a first rank product based at least in part on multiplication of a rank of the first content item in the first ranked list with a rank of the first content item in the second ranked list 450 Generating a third ranked list of the content items based at least in part on the first rank product 460 Causing at least a portion of the third ranked list of the content items to be displayed on a display

p.7This diagram shows an example architecture of a computing device.

Processing Element 504 Display Component 506 Input Device 508 Power Supply 514 Sensor 530 500 Storage Element 502 Operating System 522 Transfer App 524 Communication I/F 512 SR I/F 534 Wireless 536 Microphone 570 GPS 538 Mobile I/F 540 Wired 542

p.8This diagram illustrates an example system for sending and providing data using computing devices, networks, and a data center.

60a 62a Data Center 65 66a 68a Server 68c VM 63a RSVM Manager Network 104 64 61 67 << Gateway Router Server Manager 66b- 68b Server 68d VM RSVM 63b Manager 62b 60b

p.2A system diagram showing the components of the search query refinement system including the web server, query server, bibliographic database, daily transaction log, table generation process, and search phrase table.

WEB SERVER QUERY SERVER SEARCH PHRASE SELECTION PROCESS BIBLIOGRAPHIC DATABASE HTML DAILY TRANSACTION LOG TABLE GENERATION PROCESS SEARCH PHRASE/TABLE T-DOG WALKIN THE DOG (37) TO SAY NOTHING OF THE DOG (29) BLUEMAN DOG (28) DOG HEAVEN (22) A-DON DON GOMEZ (122) PATRICIA ANN DON (25) CASEY DON WHITE (12) DON BOX (18)

p.3A screenshot of a sample book search page with fields for author and title.

amazon.com Book Search Enter Author and/or Title Author: Exact Name Last, First Name Start of Last Name Title: Exact Title Title Word(s) Start(s) of Title Words Author Search Tips / Title Search Tips Search by Subject Subject: Exact Subject Start of Subject Subject Word(s) Start(s) of Subject Word(s) Subject Search Tips

p.4Sample log entries from a daily transaction log file, showing user identifiers, URLs, and search terms.

Friday, 13-Feb-98 02:23:52 User Identifier = 29384719287 HTTP_REFERRER= http://www.amazon.com/book_search_page PATH_JNFO=/book_search title = walking the dog items found = 2 spell_check = no Friday, 13-Feb-98 02:24:11 User Identifier = 29384719287 HTTP_REFERRER= http://www.amazon.com/book_search PATH_INFO=/ISBN = 0553562614 Friday, 13-Feb-98 06:15:03 User Identifier = 54730543261 HTTP_REFERRER= http://www.amazon.com/music_search_page PATH_INFO=/book_search artist = this and that items found = 0 spell_check = no Friday, 13-Feb-98 10:07:34 User Identifier = 027385918272 HTTP_REFERRER= http://www.amazon.com/book_search_page PATH_JNFO=/book_search subject = outdoor traile items foud = 22 spell_check = yes Friday, 13-Feb-98 11:20: 18 User Identifier = 72589100344 HTTP_REFERRER= http://www.amazon.com/book_search_page PATH_JNFO=/book_search subject = sea stars items found = 15 spell_check = no

p.5A flowchart detailing the process for generating the search phrase table.

PARSE DAILY LOG FILE TO EXTRACT QUERY SUBMISSIONS FOR WHICH ITEMS FOUND > 0 & NO SPELL CHECK USED CORRELATE SEARCH PHRASES WITH EACH KEY TERM FOUND IN THE SEARCH PHRASE (SEE FIG. 5) CREATE DAILY RESULTS FILE MERGE DAILY RESULTS FILES FOR LAST M DAYS OVERWRITE EXISTING TABLE WITH NEW TABLE

p.6A flowchart detailing the process for correlating search phrases with key terms.

REMOVE PUNCTUATION AND REPLACE WITH SPACES CONVERT TO LOWER CASE FOR EACH "KEY TERM" ADD FIELD PREFIX TO KEY TERM LOOK UP KEY TERM IN SEARCH PHRASE TABLE IS SEARCH PHRASE IN KEY TERM'S RELATED SEARCH PHRASE LIST ? INCREMENT SCORE ADD SEARCH PHRASE TO LIST AND SET SCORE NEXT KEY TERM

p.7A diagram illustrating the generation of the search phrase table from daily log files and daily results files.

Daily Log Daily Results File File Dates of Search Phrase Table Search Phrase Table

p.8A flowchart describing the selection of related search phrases from the search phrase table.

FOR EACH KEY TERM IN THE QUERY LOOK UP KEY TERM IN THE TABLE RETRIEVE THE KEY TERM'S RELATED SEARCH PHRASE LIST NEXT KEY TERM MULTI-TERM QUERY COMBINE ALL RELATED SEARCH PHRASE LISTS SELECT X SEARCH PHRASES WITH HIGHEST VALUES FILTERING OUT SEARCH PHRASES THAT ARE DIFFERENT ONLY BECAUSE OF WORD ORDER

p.9A screenshot illustrating a sample query results page showing related searches.

Your Search Results for: the title words include "DOG" Related Searches Walkin the Dog To Say Nothing of the Dog Don't Shoot the Dog Top Matches for this search: * To Say Nothing of the Dog * Don't Shoot the Dog: The New Art of Teaching & Training * Chicken Soup for the Cat & Dog Lover's Soul Full Results: The First 100 are shown below. to See more results scroll down and click the "More" button. 101 Essential Tips: Dog Care 101 Essential Tips: Training Your Dog 101 Questions Your Dog Would Ask

p.3System diagram showing the flow of data from user queries through the web server, query server, bibliographic database, and table generation process.

WEB SITE INTERNET QUERY SERVER RELATED TERM SELECTION PROCESS TABLE GENERATION PROCESS BIBLIOGRAPHIC DATABASE DAILY QUERY LOG QUERY CORRELATION TABLE

p.6Flowchart of the process for generating the correlation table.

BEGIN PARSE DAILY LOG FILE TO EXTRACT QUERY SUBMISSIONS FOR WHICH ITEMS FOUND > 0 CORRELATE TERMS BASED ON FREQUENCY OF OCCURRENCE WITHIN SAME QUERY CREATE DAILY RESULTS FILE MERGE DAILY RESULTS FILES FOR LAST M DAYS OVERWRITE EXISTING CORRELATION TABLE WITH NEW CORRELATION TABLE END

p.7Sample query correlation table mapping before a query is added, showing key terms and their related terms.

S-BIKE A-CARLSON (2) S-EXCERCISE (12) A-FRANKLIN (5) S-OUTDOOR S-TRAIL T-BIKE (73) T-DINING (100) T-EDUCATION (36) S-SPORTS (41) S-TRAIL (65) S-OUTDOOR T-DINING S-BLAZING T-BLAZING (5) S-BIKE (63) A-GARRETT (21) S-MIX (92) S-OUTDOOR (23) T-TRAIL (7) S-SPORTS (12) S-VACATION (9)

p.10Flowchart illustrating the process for selecting related query terms from the correlation table.

BEGIN FOR EACH TERM IN THE QUERY LOOK UP TERM IN THE CORRELATION TABLE RETRIEVE THE TERM'S RELATED TERMS LIST NEXT TERM MULTI-TERM QUERY YES NO COMBINE ALL RELATED TERMS LISTS SELECT X TERMS WITH HIGHEST VALUES END

p.12Example of a search results page with related query terms.

amazon.com Book Search Your Search Results for: the subject words include "OUTDOOR TRAIL" Related Query Terms: Click on a link below to narrow your search by adding a related term. OUTDOOR TRAIL - BIKE OUTDOOR TRAIL - SPORTS OUTDOOR TRAIL - VACATION Top Matches for this search: * Outdoor & Trail Guide to Yosemite * Outdoor Training on Mountian Bike Trails

p.2This image shows example displays of search results presented in response to the query 'dresses'.

dresses Party Dress by Acme $74.99 Block Dress by ClosetPop $52.50 Short Stripes by HappyClothes $87.99 formal dresses Party Dress by Acme $74.99 Block Dress by ClosetPop $52.50 Block Dress by ClosetPop $52.50

p.3This image shows a diagram illustrating the search results that can be displayed based on query attributes.

Too formal 204 Initial image More formal 206 Not categorized as formal 208 Categorized as formal but not more formal 210

p.3This image illustrates examples of semantic relationships that can be learned.

waitress woman drinking eating drank ate waiter man Gender Verb tense Ohio Columbus California Sacramento Washington Olympia Colorado Denver State-Capital

p.4This image depicts text embedding models for cross-media similarity computation.

BoW(q) Φ(Χ) beautiful wedding image word matrix p(w/x) label matrix text pooling pooling CNN word2vec CNN image Φ(Χ) word matrix s(q) h₁(q) text pooling σ(W1*s(q)+b1) (W2*h1(q)+b2) CNN image

p.5This diagram shows semantic similarities learned in accordance with embodiments.

O four O quatro O uno O five O one O cinco O two O three O tres O dos

p.5This image illustrates an example approach to determining relevant results in a multi-dimensional semantic space.

Attribute a2 %%% Attribute a1 Object Class

p.6This image illustrates an example parsing and/or semantic analysis that can be performed on a received query in accordance with various embodiments.

shoes for a wedding for my daughter shoes for a wedding for my daughter shoes wedding daughter dress designer formal gown ceremony female born: 1999 style:arty color:dark single

p.7This image illustrates an example system that can be used to implement aspects of the various embodiments.

Content Provider Environment Network Third Party Provider Interface Content Query Parser Content server Search Engine Natural Language Data Dialog Manager User CNN/GAN Training Component

p.8This image illustrates an example process used to determine content based on query analysis.

Receive query Analyze query to determine primary object, query attributes, and intent Generate query vector for primary object and intent Modify query vector using determiend attributes Confident results? Yes No Return content for at least a subset of the matches as result(s) Determine clarifying attribute(s) Prompt user to provide information for attribute(s) Receive? Yes No Provide content for closest matches as results

p.10This image illustrates a computing device.

Memory Communication Component Processor Input Device Display

p.6A diagram of the recommendation system architecture.

WEB SITE EXTERNAL COMPONENTS RECOMMENDATION SERVICE COMPONENTS BOOKMATCHER PROCESS OFF-LINE TABLE GENERATION PROCESS (FIG.3) RECOMMENDATION PROCESS (FIGS.2,5&7) SIMILAR ITEMS TABLE (ITEM-TO-ITEM MAPPINGS) USER PROFILES • PURCHASE HISTORIES • RECENTLY VIEWED ITEMS • ITEM RATINGS • SHOPPING CART CONTENTS • RECENT SHOPPING CART CONTENTS POPULAR ITEM SIMILAR ITEMS LIST PRODUCT_ID (PRODUCT_ID, CI), (PRODUCT_ID, CI),... PRODUCT_ID (PRODUCT_ID, CI), (PRODUCT_ID, CI),... N ITEMS FIG. 1

p.7Flowchart of the general process for generating personalized recommendations.

GENERATE PERSONAL RECOMMENDATIONS IDENTIFY ITEMS KNOWN TO BE OF INTEREST TO USER RETRIEVE SIMILAR ITEMS LIST (IF ANY) FOR EACH ITEM OF KNOWN INTEREST WEIGHT SIMILAR ITEMS LIST(S) (OPTIONAL) COMBINE SIMILAR ITEMS LISTS IF MULTIPLE LISTS SORT RESULTING LIST FROM HIGHEST- TO-LOWEST SCORE FILTER SORTED LIST TO GENERATE RECOMMENDATIONS LIST ADD ITEMS TO RECOMMENDATIONS LIST (OPTIONAL) RECOMMEND TOP M ITEMS FROM RECOMMENDATIONS LIST FIG. 2

p.8Flowchart of the process for building the similar items table using purchase histories.

BUILD SIMILAR ITEMS TABLE RETRIEVE PURCHASE HISTORIES FOR ALL CUSTOMERS GENERATE TEMPORARY TABLE MAPPING CUSTOMERS TO PURCHASED ITEMS GENERATE TEMPORARY TABLE MAPPING PURCHASED ITEMS TO CUSTOMERS IDENTIFY POPULAR ITEMS FOR EACH (POPULAR_ITEM, OTHER_ITEM) PAIR, COUNT NUMBER OF CUSTOMERS IN COMMON COMPUTE COMMONALITY INDEXES SORT OTHER_ITEMS LISTS FILTER OTHER_ITEMS LISTS TRUNCATE OTHER_ITEMS LISTS AND STORE IN TABLE CUSTOMER USER_A USER_B ITEM ITEM_A ITEM_B OTHER_ITEM POPULAR_ITEM POPULAR_A POPULAR_B FIG. 3A

p.9Flowchart of the process for building the similar items table using query log records.

BUILD SIMILAR ITEMS TABLE RETRIEVE QUERY LOG RECORDS GENERATE TEMPORARY TABLE MAPPING SESSIONS TO VIEWED ITEMS GENERATE TEMPORARY TABLE MAPPING VIEWED ITEMS TO SESSIONS IDENTIFY POPULAR ITEMS FOR EACH (POPULAR_ITEM, OTHER_ITEM) PAIR, COUNT NUMBER OF SESSIONS IN COMMON COMPUTE COMMONALITY INDEXES SORT OTHER_ITEMS LIST FILTER OTHER ITEMS LIST TRUNCATE OTHER_ITEMS LISTS AND STORE IN TABLE SESSION SESSION_A SESSION_B ITEM ITEM_A ITEM B OTHER_ITEM POPULAR_ITEM POPULAR_A POPULAR_B FIG. 3B

p.10Venn diagram illustrating purchase relationships for different item sets.

ITEM_P ITEM_X 300 300 25 ITEM_Y 30,000 FIG. 4

p.11Flowchart of the process for generating instant recommendations.

GENERATE INSTANT RECOMMENDATIONS IDENTIFY ALL POPULAR ITEMS PURCHASED OR RATED BY USER WITHIN LAST SIX MONTHS RETRIEVE SIMILAR ITEMS LISTS FROM TABLE WEIGHT EACH SIMILAR ITEMS LIST BASED ON USER'S PURCHASE DATE OR RATING OF CORRESPONDING POPULAR ITEM MERGE SIMILAR ITEMS LISTS (IF MULTIPLE LISTS) WHILE SUMMING SCORES SORT RESULTING LIST FROM HIGHEST-TO-LOWEST SCORE FILTER RESULTING LIST BY DELETING ITEMS WHICH HAVE BEEN PURCHASED, HAVE BEEN RATED, HAVE A NEGATIVE SCORE, OR FALL OUTSIDE DESIGNATED PRODUCT GROUP OR CATEGORY OPTIONALLY SELECT ITEM FROM USER'S RECENT SHOPPING CART CONTENTS AND INSERT INTO ONE OF THE TOP M POSITIONS IN LIST RECOMMEND TOP M ITEMS FROM LIST FIG. 5

p.12Screenshot of a web page display format for listing book recommendations.

amazon.com Instant Recommendations Hello, John Gerry. We think you'll like these items in All Categories • The Other Side of Midnight; Sidney Sheldon • Inside Intel; Tim Jackson • The Road Ahead: Bill Gates, et al • The Doomsday Conspiracy; Sidney Sheldon • Skinny Legs and All; Tom Robbins More Recommendations Already own any of these titles? Know you don't like one? Refine your recommendations and we'll immediately show you new choices! New! We have music recommendations for you! Amazon.com Home | Shopping Cart | Your Account | Recommendation Center FIG. 6

p.13Flowchart of the process for generating shopping cart based recommendations.

GENERATE SHOPPING CART BASED RECOMMENDATIONS FOR EACH SHOPPING CART ITEM THAT IS A POPULAR ITEM, RETRIEVE SIMILAR ITEMS LIST FROM TABLE MERGE SIMILAR ITEMS LISTS WHILE SUMMING CI VALUES SORT RESULTING LIST FROM HIGHEST TO LOWEST SCORE FILTER RESULTING LIST BY DELETING ITEMS THAT ARE CURRENTLY IN THE SHOPPING CART AND ITEMS THAT HAVE BEEN PURCHASED OR RATED RECOMMEND TOP M ITEMS FROM LIST FIG. 7

p.14High-level data flow diagram for displaying session based recommendations.

WEB SITE EXTERNAL COMPONENTS RECOMMENDATION SERVICE COMPONENTS SESSION-SPECIFIC ITEMS OF KNOWN INTEREST SESSION RECOMMENDATIONS PROCESS SIMILAR ITEMS TABLE OFF-LINE TABLE GENERATION PROCESS (ITEM-TO-ITEM MAPPINGS) PRODUCT VIEWING HISTORIES QUERY LOG STORAGE CLICK-STREAM TABLE (CACHE MEMORY) HTTP/XML APPLICATION FIG. 8

p.15Click stream table for tracking user sessions.

CLICK STREAM TABLE SESSION ID DETAIL PAGE LIST BROWSE NODE LIST SEARCH QUERY LIST 1234567 dp1, dp2, dp3, dp4 bn1, bn2, bn3, bn4 "a space odyssey","isaac asimov" FIG. 9

p.16Page to item lookup table.

PAGE-ITEM TABLE PAGE IDENTIFICATION ITEM IDENTIFICATION DP1 "2001: A Space Odyssey" BN1 "Sony DVP-S360 DVD Player", "Toshiba SD- 1600 DVD player," FIG. 10

p.17Screenshot of the personalized web page.

amazon.com WELCOME DIRECTORY VIEW CART WISH LIST YOUR ACCOUNT HELP TODAY'S FEATURED STORES SOFTWARE ELECTRONICS TOOLS & HARDWARE INTERNATIONAL ►TOP SELLERS FRIENDS & FAVORITES ►FREE E-CARDS ►TAX CENTER SEARCH: All Products GO BROWSE: BOOKS THE PAGE YOU MADE BY YOU, FOR YOU, IN REAL TIME! Welcome, Bob Smith, to the Page You Made! (If you're not Bob Smith, click here.) The Page You Made is based on your recent clicks on our site. Our goal is to help you find what you want and discover related items. You can change this page by visiting more product pages or removing items from the list of recently viewed items. Learn more My Recently Viewed Items Top Sellers Children's Books > Baby-3 > Board Books Moo, Baa, LA LA LA (Boyton, Sandra. Boynton Board Books.) by Sandra Boynton (Illustrator), Kate Klimo (Editor) The Going to Bed Book (Boynton Sandra. Boynton Board Books.) by Sandra Boynton, Kate Kilmo (editor) Update page De-Select all items 1. Dr. Seuss's ABC An Amazing Book (Bright and Early Board Book) by Seuss Our Price: $3.99 You Save: $1.00 (20%) 2. There's a Wocket in My Pocket! : Dr. Seuss's Book of Ridiculous Rhymes by Seuss, et al Our Price: $3.99 You Save: $1.00 (20%) 3. Goodnight Moon by Margret Wise Brown, Clement Hurd (Illustrator) Our Price: $6.36 You Save: $1.59 (20%) Recommendations Brown Bear, Brown Bear, What Do You See? From Horn Book Good Night Gorilla by Sandra Boynton (VHS) Baby Bach Video From Baby Einstein Company FIG. 11

p.18Screenshot of items you may also be interested in.

amazon.com WELCOME DIRECTORY ►TOP SELLERS ►FREE E-CARDS ►TAX CENTER International SEARCH: All Products GO! Intel Pocket Concert 128MB Digital Audio Player Other products by Intel Add to shopping cart Customers who viewed this item also viewed these items: • Intel Pocket Concert 128MB Digital Audio Player & Accessory Kit by Intel $321 • lomega HipZip Digital Audio Player by lomega $266 • 12Go eGo 340MB MP3 Player (Blue) by 12go.com $499 • CreativeLabs N640001 NOMAD II Digital Audio Player by Creative Labs $199 View more FIG. 12

p.19Flowchart for generating related items list.

GENERATE RELATED ITEMS LIST FOR DISPLAY ON DETAIL PAGE OF "POPULAR” PRODUCT RETRIEVE SIMILAR ITEMS LIST FOR PRODUCT OPTIONALLY FILTER SIMILAR ITEMS LIST TO REMOVE PRODUCTS FALLING OUTSIDE PRODUCT CATEGORY OF "POPULAR" PRODUCT DISPLAY N TOP-RANKED PRODUCTS FROM FILTERED OR UNFILTERED LIST FIG. 13

p.2A diagram illustrating stochastic label aggregation used in a multi-objective ranking system.

Online Retailer 114 Objective(s) 122 112 104 110 Training Dataset 128a Training instance 1 102 Multi-objective ranking system 102 Training Dataset 128b Training instance 1 Labels 130a 1,2,... N : Training instance N Stochastic label aggregation function 140 Probability distribution 142 Label 130b b1 : Training instance N Labels 132a 1,2,... N FIG. 1 Label 132b l2

p.3A table comparing different search rankings resulting from different optimization techniques, including single-objective optimization and MORO using stochastic label aggregation.

Online Retailer 114 Objective(s) 122 112 104 Training Data 220 102 Multi-objective ranking system 102 Single Objective Optimization (Purchase) MORO using stochastic label aggregation Ranking 230 Ranking 240 Pur- Non- Pur- Non- Pur- Non- chase Quality defect chase Quality defect chase Quality defect 110 Item 1 0 0 0 Item 2 1 0 1 Item 2 1 0 1 Item 2 1 0 1 Item 1 0 0 0 Item 3 0 1 1 Item 3 0 1 1 Item 3 0 1 1 Item 1 0 0 0 NDCG 0.63 0.5 0.69 NDCG 1 0.5 0.92 NDCG 1 Purchase NDCG optimal 0.63 Purchase NDCG Optimal, Non-defect NDCG optimal, Quality NDCG improved 1 FIG. 2

p.4A block diagram illustrating a two-phase combination of ranking models, where the ranking algorithm combines models optimized for different objectives and generates a ranked list.

Online Retailer 114 Objective(s) 122 112 104 Phase 1 Model l1 110 Model l2 Me1(q, p) Mez(q, p) Ranking algorithm 301 FIG. 3 102 Multi-objective ranking system 102 Phase 2 Ranking model 302 (Optimized w.r.t. aggregated label) Ranked list 304

p.5A scatter plot depicting the costs for two different objectives for various different ranking models, including Pareto frontiers.

Cost 1.0 0.8 0.6 0.4 0.2 * 0.8.0 M\Par(M) Par(M) \Par(M/M) Par(M/M) Par(M) CD 0.2 0.4 0.6 Cost 0.8 1.0 Pareto frontiers: Par(M), Par(M*) and Par(M\M*) FIG. 4A

p.6A plot of costs for two different objectives for stochastic label aggregation and deterministic label aggregation methods.

0.0 0.4 0.2 0.6 0 70 60 90 80 7 1.0 1.2 Cost aob, cod adb,c>d M Par(M) Mdet abb, cod ab, c▷d 0.00 0.01 0.02 0.03 0.04 0.05 0.06 Cost Proposition 4: deterministic vs stochastic label ag- gregation costs FIG. 4B

p.1This is a diagram of a system for reformulation of tail queries for product searches.

160 -100 120 114- 110- SEARCH SERVICE TAIL QUERY PLATFORM DEVICES SEARCH RESULTS 138 114- TAIL QUERY HEAD QUERY 134 150- 154 130 MAPPING SUBSYSTEM CONSTRUCTOR MODEL SUBSYSTEM 140 MODEL REPOSITORY 180 HEAD QUERY DATA 170 ENGAGEMENT ACTIVITY DATA TAIL QUERY DATA FIG. 1

p.3This is a system diagram to generate a mapping model.

210 230 DATA CONSTRUCTOR UNIT EMBEDDING GENERATOR UNIT 240 220- CLASSIFIER UNIT LABELED DATA STORE 250 BANDIT ►GENERATOR UNIT FIG. 2 150

p.4This is a schematic diagram of a Siamese transformer network.

350- 300 1/0 CROSS ENTROPY 340(1) 340(2) 1 DENSE LAYER (256-d) DENSE LAYER (256-d) 330(1) 330(2) FEED FORWARD SELF ATTENTION FEED FORWARD SELF ATTENTION 320(1) 320(2) FEED FORWARD SELF ATTENTION FEED FORWARD SELF ATTENTION 310 310 GLOVE (64x6-d) GLOVE (64x6-d) 305(1) 305(2) 91 Q2 FIG. 3

p.5This algorithm is for updated PIE versions using pseudo-oracle.

400 Example Algorithm - Updated PIE versions using pseudo-oracle Input: Initial model PIE), iterations I, source query batches S1, S2, ... S₁, target query set H, k, pseudo-oracle f(-) 1: for i = 1, 2, ..., I do 2: Get embedding h using PIE;-1 for all he H for source query s ∈ S₁ C S do Get embedding s using PIE-1 3: Set D₁ = $ (empty dataset) 4: 5: Obtain k nearest neighbors h.....h of s 6: 7: 8: 9: 10: Obtain rewards from pseudo oracle y = f(s,h) Vj ∈ [k] S Append {(s, h, y¹), ..., (s, h, y)} in Di end for Finetune PIE/-1 on D₁ to get PIE; with early stopping 11: end for 12: Output: Models PIE1, PIE2, ..., PIE1. FIG. 4

p.6This diagram illustrates persistent noise in a labeling function based on a machine-learning (ML) product-category classifier.

510 f(,) Boundary Error 512 515 520 True Boundary 514 514 FIG. 5

p.7The algorithm is cBLIP for Query Reformulation.

Example Algorithm - cBLIP for Query Reformulation Input: Parameters ẞ > 0, μο € Rdxd, στεR 2 0 1: for t = 1, 2, ..., T do 2: Sample W₁ ~ N(µt-1,021) 3: Observe context s₁ (a source query) 4: Choose optimal action h₁ = arg maxheth Wist h Wast 5: Observe reward by sampling rt β 6: Set 82 = β2 7: Set με = -1 + + (ht ©ht)1(st ν T2 σ 1-1 St) T δ [his] 002 -1] 8: 2 2 2 Set o² = 2-1 1-0 (1-1) [(?)71dxds?]]; στ δ where v(z) = Niz:0.1) and a(z) = v(z)(v(z) + z). 9: end for (2:0,1) 10: Output: μ. 02. For inference, final matrix W = p FIG. 6 600 000

p.8This is a graph of query reformulation performance.

Comparison of Query Reformulation Performance Cumulative % Improvement in PQRC 40 35 30 725 20 15 10 5 0 QTD AvgGlove PIE0 PIE3 PIE2 PIE2-CBLIP-PTB Reformulation Models FIG. 7

p.2A diagram illustrating the system architecture for responding to user queries based on user intent.

(FOR EXAMPLE) OPERATIONS CENTER 104 QUERY TRANSACTION LOG TABLE 128 124 QUERY MODULE 116 FUNCTION SELECTION MODULE 136 SEARCHABLE STORE(S) 120 FUNC- FUNC- QUERY-RESULT LOGGING MODULE 126 CLASSIFICATION DETERMINATION MODULE 130 FUNC- 140 B 122 Π DATA MINING OTHER DATA 132 MODULE 118 USER INTERACTION MODULE 134 COUPLING MECHANISM (E.G., INTERNET) QUERY RESULTS 106 REPRESENTATIVE USER DEVICE 102 INPUT UNIT 112 PROCESSING UNIT 108 PRESENTATION UNIT OPTIONAL CLIENT-SIDE SEARCHING FUNCTIONALITY 144 USER INTERFACE 114 100

p.3A diagram illustrating the query module with ranking functions.

PRIOR CLASSIFICATIONS REAL- TIME FUNCTION SELECTION MODULE SEARCHABLE STORE(S) 120 136 RANKING FUNCTION A (DEFAULT) 204 RANKING FUNCTION B 206 RANKING FUNCTION n 208 SEARCHING MODULE 210 RESULT RANKING FUNCTIONS 202 RANKED SEARCH RESULTS 214 RAW SEARCH RESULTS 212 USER INTERACTION MODULE 134 QUERY MODULE 116

p.4An example of a query transaction log, showing queries and user selections.

QUERY TRANSACTION LOG 128 QUERY "COUNTRY-¬ 302 MUSIC" "CRYING"304 USER SELECTION (E.G., PURCHASE) "COUNTRY MUSIC GREATEST HITS I" (1) "BACKROADS COUNTRY MUSIC" (5) 308 "BEST OF HANK WILLIAMS" (1223) "30 YEARS OF GOLD" BY GEORGE JONES (1334) 306 "MAN IN BLACK" BY JOHNNY CASH (1367) 310 "GREATEST HITS" BY WILLY NELSON (2004) "SAD SONGS FOR A RAINY DAY" (2) 314 "COLLECTED BLUES: SAD DAYS AHEAD" (4) 312 "THE SKY IS CRYING" S. R. VAUGHAN (1003) "CRYING" BY ROY ORBISON (1025) 316 "THE CRYING GAME" SOUNDTRACK (2002)

p.4An example of a query classification table.

QUERY CLASSIFICATION TABLE 124 QUERY CLASSIFICATION "COUNTRY MUSIC THEME-BASED QUERY 406 "CRYING" 404 TITLE-BASED QUERY 408

p.5A diagram showing user interaction and theme based function selection.

Exemplary Input Interface Please specify what item you want "Country Music" THEME-BASED RANKING FUNCTION 506 RANKED SEARCH RESULTS 508 Exemplary Query Results Willy Nelson's Greatest Hits ADD TO CART Waylon Jennings Lonesome in El Paso ADD TO CART Johnny Cash Man in Black ADD TO CART ETC.

p.6Illustrative diagram of processing functionality.

UI PRESENTATION SCREENS 114 PRESENTATION UNIT 110 INPUT DEVICE(S) 112 USER PROCESSOR STORAGE 608 602 PROCESSING UNIT 108 I/O 612 614 INTERFACE 610 RAM ROM 604 606 COUPLING MECHANISM 106

p.7A flowchart describing a procedure for collecting query-related behavior.

START RECEIVE AND STORE USER'S QUERY 702 PROVIDE QUERY RESULTS TO THE USER 704 RECEIVE AND STORE INDICATION OF USER'S RESPONSE TO QUERY: • CLICK-THROUGH • CART SELECTION • PURCHASE • Етс. 706 REPEAT FOR OTHER QUERIES BY OTHER USERS END

p.8A flowchart describing a procedure for classification analysis of query data.

START ACCESS QUERY TRANSACTION LOG 802 FOR EACH QUERY, DETERMINE CLASSIFICATION 804 STORE CLASSIFICATION IN CLASSIFICATION TABLE 806 REPEAT PERIODICALLY OR IN RESPONSE TO OTHER TRIGGERING EVENT END

p.9A flowchart describing the steps in responding to a user query based on intent.

START RECEIVE USER'S QUERY 902 DETERMINE CLASSIFICATION CORRESPONDING TO QUERY 904 GENERATE RAW SEARCH RESULTS 908 GENERATE QUERY RESULTS BASED ON RESULT-GENERATING FUNCTION ASSOCIATED WITH CLASSIFICATION 906 END APPLY RANKING FUNCTION TO RAW QUERY RESULTS TO GENERATE QUERY RESULTS 910

p.1Example display of content on a computing device screen showing search results for "knee length blue dress" and a list of results.

knee length blue dress Party Dress by Acme S74.99 Block Dress by ClosetPop $52.50 Short Stripes by HappyClothes $87.99

p.1Example search result with attribute and confidence values shown.

90% blue 85% solid 78% full length sleeve 65% above the knee 88% blue 92% solid 73% full length sleeve 72% above the knee 91% blue 78% solid 82% full length sleeve 55% above the knee 90% blue 84% solid 78% full length sleeve 64% above the knee

p.2A system diagram showing the content provider environment and its interactions with a client device and third-party provider.

Content Provider Environment Interface Content server Style Manager User Style Training Component Models CNN/GAN Localizer Image Analyzer Classifier Network Third Party Provider

p.3Illustrations of an image of a person and how regions of the image can be selected for analysis.

Get Style

p.4A flowchart illustrating the process of locating content for similar items.

Receive query image including a representation of an item Analyze the query image to identify a type of the item Determine a set of attributes exhibited by the item in the query image Determine a set of confidence values for the attributes Compare the attributes and the confidence values against previously determined data stored for items of that type Determine a set of similar items based at least in part upon the comparison Locate content for the similar items Provide the content for at least a subset of most similar items in response to the query image

p.5A flowchart illustrating the process of identifying stylistically similar items.

Obtain image data including a representation of at least one item of interest Perform localization on the image data to determine one or more regions corresponding to potential items Perform classification on the region(s) to determine an item type for each region Select a region to analyze Select trained machine learning model for item type Process image data for the region using the trained model to obtain a set of attributes and corresponding confidence values More regions? No Yes Identify stylistically similar items using the attributes and confidence values

p.6A system diagram showing the client device interacting with the multi-tenant resource provider environment.

Client Device Network Resource Manager Acct. Resource Data

p.7A block diagram illustrating the components of a computing device.

Wireless Components Power Components Input Element Memory Processor Display Imaging Element Orientation Determining Element Positioning Element

p.2This diagram illustrates acquiring an item of interest using a portable computing device with a camera.

110 104 108 R 122 k 102 106 104 FIG. 1(a) FIG. 1(b) 124 100 120

p.3This diagram illustrates example approaches for removing background from an image.

202 222 FIG. 2A 200 210 212 FIG. 2B 220 230 234 FIG. 2C FIG. 2D 232

p.4This diagram illustrates stages of an example process for determining a torso region and/or upper body region.

202 322 342 300 322 FIG. 3A FIG. 3B 340 362 FIG. 3C FIG. 3D 320 360

p.5This diagram illustrates stages of an example process for determining a clothing region of an image.

400 420 FIG. 4A FIG. 4B 440 462 FIG. 4C FIG. 4D 460

p.6This diagram illustrates an example approach to locating specific items represented in an image.

FIG. 5A FIG. 5C FIG. 5E 500 502 520 524 522 540 544 542 516 518 512 514 FIG. 5B FIG. 5D FIG. 5F 510 530 532 550 552

p.7This diagram illustrates stages of an example process for utilizing a probability map.

362 600 362 342 342 462 462 222 642 FIG. 6A 640 222 662 FIG. 6B FIG. 6C FIG. 6D 620 660

p.8This diagram illustrates an example of image similarity-based group browsing.

702 700 722 724 726 703 708 Length 710 Pattern 742 744 746 0000 706- Color Cut 732 734 736 704- FIG. 7 720 730 740

p.9This diagram illustrates an example of image similarity-based group browsing, highlighting color as a chosen attribute.

702 800 722 742 736 734 726 746 732 744 724 703 820 Color FIG. 8 736 734 732 830 810

p.10This diagram illustrates an example of a search results interface.

900 Match Results 736 Party dress by Acme $74.99 734 Spring Dress by Shirley $52.50 732 Touch of Grey by Designee $27.99 FIG. 9 910

p.11This diagram illustrates an example process for image similarity-based group browsing.

1002 1004 1006 Receive query image ↓ Obtain visual attribute categories and visual attributes Analyze query image 1008 Obtain items in item catalog 1010 1012 ↓ Determine visual similarity score for items in catalog Generate visual similarity result set 1014 Receive selection of attribute 1012 Determine product group Yes 1014 -<Non-matching Groups?> No 1000 1016- 1018 Generate listing for presentation Remove items of non-matching groups FIG. 10

p.12This diagram illustrates an example process for image similarity-based group browsing using explicit and implicit attributes.

1102 1104 1106 1108 1110 1112 1114 1116 Determine item of interest ↓ Determine explicit visual attributes ↓ Select set of visually similar items Determine subsets of items Determine ranking of items Determine implicit visual attributes Determine implicit visual similarity score Determine new ranking of items FIG. 11 1100

p.13This diagram illustrates an example categorization tree.

C 1202 1204 1206 1208 FIG. 12 1200

p.14This diagram illustrates an example system for performing image similarity-based group browsing.

1302- Network 1304 -1324 1300 1306 1308- Network Interface Layer 1310- Matching Service 1316 1312 1314 Log Identification Service Similarity Service Match Product Human 1318 1322 1320 FIG. 13

p.15This diagram illustrates an example computing device.

1402 1404 1406 FIG. 14 Memory 1504 Communication Component 1508 Processor 1502 Display 1506 Input Device FIG. 15 1510 1400 1500

p.16This diagram illustrates an example environment.

1602 1604 Web Server Network 1606 1608 1000000 Application Server Content Session User Information 1612 1614 1616 1610 FIG. 16 1600

p.8This diagram depicts the steps involved in receiving a query image, determining its feature vector, identifying attribute manipulation options, and providing content for similar items.

Receive query image Determine feature vector of query image using trained neural network for attributes Determine attribute labels associated with feature vector Identify attribute manipulation options for attribute labels Provide attribute labels, attribute manipulation options, and content for highest-ranked items having similar visual attributes Receive one or more attribute manipulations Determine attribute manipulation inverse matrix associated with manipulated visual attributes Determine manipulated feature vector using attribute manipulation inverse vector matrix Determine location for manipulated feature vector in attributes space Determine subset of items with representations near manipulated feature vector location Provide content for at least a number of highest-ranked items as having similar visual attributes

p.5This diagram illustrates the visual attribute label groupings using a multi-dimensional, multiple attribute trained neural network.

Multi-Dimensional, Multiple Attribute Trained Neural Network Visual Attribute Label Grouping Visual Attribute Label Grouping Visual Attribute Label Grouping

p.6This image displays the manipulation of feature vectors by showing original feature vector, inverse matrix associated with attribute manipulation and finally resulting manipulated feature vector.

Original Feature Vector Inverse Matrix Associated with Attribute Manipulation(s) Manipulated Feature Vector

p.3This diagram illustrates the search engine system and its components.

INTERNET WEB CRAWLER PRODUCT SCORE GENERATOR INDEX TOOL WEB SERVER QUERY LOG PRODUCT SPIDER/DATABASE KEYWORD URL TITLE SQUIB SCORE QUERY SERVER COSMOS www.abc.com www.def.com www.geh.com CATEGORY SPELL CHECKER RANKING PROCESS SEARCH TOOL COSSACK www.mno.com www.pqr.com www.stu.com : : BOOKS MUSIC VIDEO AUCTIONS SOFTWARE ELECTRONICS

p.4This image shows a sample search tool interface page.

SEARCH BOOKS MUSIC VIDEO AUCTIONS ALL PRODUCTS Mark Twain WELCOME TO amazon.com BROWSE BOOKS Bestsellers, Computers Kids, Business MUSIC Top Sellers, New Releases Soundtracks VIDEO DVD's, Top Sellers New Releases AUCTIONS How Auctions Work Collectible, Sports

p.5A search results page is displayed for the search query "Mark Twain".

BOOKS Letters from the Earth Following the Equator: A Journey Around the World Joan of Arc See all matching results in Books VIDEOS Mark Twain Tonight (1967); VHS The Adventures of Mark Twain (1944); VHS Mark Twain (1995) A&E Biography; VHS See all matching results in Videos... AUCTIONS 10 Takes from Mark Twain on CDROM current bid: $9.99 Mark Twain and the Laughing River, Audio CD current bid: $7.99 Mark Twain: The Musical (1991) See all matching results in Auctions... MUSIC See all matching results in Music... Additional Matches for Mark Twain from other on-line merchants: SOFTWARE A Horse's Tale Extracts from Adam's Diary A Visit to Heaven See all matching results in Software... ELECTRONICS See all matching results in Music...

p.7This diagram shows the process of generating the product spider database.

WEB CRAWL A PORTION X OF THE WORLD WIDE WEB APPLY PAGE ANALYZER SUBMIT URL'S FROM PREVIOUS PRODUCT SPIDER DATABASE FOR RECHECKING PAGE SATISFY FILTER NO DISCARD YES SUBMIT URL TO WEB CRAWLER FOR RECHECKING WEB CRAWL SUBMITTED PAGES APPLY PAGE ANALYZER PAGE SATISFY FILTER NO DISCARD YES APPLY INDEX TOOL

p.8The diagram shows the process for generating the return page for an "All Products" search query.

PROMPT FOR ALL PRODUCTS QUERY RECEIVE SEARCH QUERY APPLY THE QUERY TO ALL CATEGORIES RETURN QUERY RESULTS FROM EACH CATEGORY TO QUERY SERVER DETERMINE A RELEVANCE RANKING FOR EACH COMPETING CATEGORY ARRANGE COMPETING CATEGORIES FOR DISPLAY GENERATE SEARCH RESULTS PAGE

p.9This diagram shows data structures for a "Books" database.

BOOKS DATABASE BOOKS FULL TEXT INDEX KEYWORD ITEM IDENTIFIER TWAIN 1311302165 2561203113 3742073036 4131415926 4603283881 4843044180 6271824119 7944285958 TWANG 3674063281 8883452918 TWEAK 2319171311 6306872726 BOOKS POPULARITY SCORE TABLE KEYWORD ITEM IDENTIFIER POPULARITY SCORE MARK 1311302165 1 2561203113 22 3742073036 7 :: TWAIN 1311302165 3 2561203113 41 3742073036 313

p.10This diagram shows the category ranking from category popularity scores.

FOR EACH SET OF COMPETING CATEGORIES FOR EACH CATEGORY DETERMINE A CATEGORY POPULARITY SCORE FROM CONSTITUENT SEARCH RESULT ITEMS NEXT CATEGORY CREATE A CATEGORY RANKING FROM THE CATEGORY POPULARITY SCORES NEXT SET

p.11This diagram shows processing uncommon search queries.

SEARCH AUCTIONS DATABASE SEARCH POPULARITY SCORE TABLES ANY RESULTS YES NO SEARCH FULL TEXT INDEXES ANY RESULTS YES RUN SPELL CHECK REPEAT BOXES 910-935 ANY RESULTS YES REPEAT BOXES 910-935 WITH SINGLE TERM QUERIES ANY RESULTS