Comparing User-Assigned Terms
with Indexer-Assigned Terms
for storage and retrieval of moving images:

• Research to date in the realm of indexing image documents for purposes of storage and retrieval has been concentrated in the area of collections of art. Probably this is due to the importance of these collections as a cultural resource, and to the long-recognized value of art collections. However, non-art images are also collected, and they are associated with a different set of indexing problems. Within this second area are moving image documents which need to be indexed at the shot level, such as are found in television archives and stockshot collections for both film and video.
  
  
• A study was conducted to gather evidence which would suggest appropriate theoretical directions for providing subject access to still and moving non-art images in storage and retrieval systems (Turner 1994). Participants screened a videocassette containing stock footage and for each shot supplied words or phrases which they felt were obvious keys for retrieval at a later date by themselves or others. Analysis was conducted on the most popular terms supplied for each of the shots on the tape. One of the significant findings was that across all the 181 participants and all 44 shots, on average almost 60% of the participants named the most popular term for each shot. Another important finding of the study was that participants overwhelmingly supplied pre-iconographical ( ofness ) terms rather than iconographical ( aboutness ) terms, for the most part simply naming the objects and events they saw in the shots.
  
  
• The present paper is concerned with a follow-up study in which the most popular terms supplied by participants in the study were compared with the terms assigned by professional indexers in the source stockshot collection in order to discover the degree of correspondence. If there is agreement, then indexers are making access points available which correspond to user needs.

• The stockshot collection at the National Film Board of Canada contains over 150 000 shots, for the most part stored in the original negative form. Information concerning 18 000 of these shots is stored in an online database, in use since 1988, and information about the remaining 132 000 shots is stored in a card file, started in 1941 and closed in 1988. The sample taken for the original study was representative of the entire collection. Records for 11 of the shots came from the computer database, and the remaining 33 records came from the older card file.
  
  
• Since the level of indexing is specific in the database (naming significant people, objects and events seen in the image) and more general in the card file (giving the equivalent of subject headings in library catalogues), direct comparison of the indexing is not possible. Interestingly, however, the significant people, objects and events recorded in the card file indexing are often named as subject headings, reinforcing the idea that in practice it is difficult to separate the pre-iconographical and iconographical levels of description in subject indexing (Turner 1994, 151-53).
  
  
• The first part of the present study is concerned with whether the most popular term for any given shot is included as in indexing term or part of an indexing term either in the stockshot database or the card file of the source collection. The second part is concerned with whether the most popular term is also included in the written running description of the content of each shot which forms part of the catalogue record.

• The most popular terms for all the shots were compiled and this list was used as the primary reference for comparison with information found in the database and in the card file. Identifying the most popular term for each shot is not straightforward because for most shots, more than one term achieved the highest score. Of the 44 shots used in the study, 17 had a single most popular term; 16 shots had two terms competing for the distinction; 11 shots had 3 or more terms competing. The greatest number of terms competing for the distinction was 9 (one shot). Any of the terms which achieved the highest score for each shot was used for comparison against the data.
  
  
• In the original study, the terms were analyzed using the first significant word of each descriptor supplied, in truncated form, following a method used by Furnas, Landauer et al. (1983) and based on the idea that in a computer retrieval system the search would probably be conducted in this way. However, the technique of collapsing terms to the first significant word was not strictly adhered to in order to keep concepts separate when verbal expressions are identical, and to deal with variant spellings, illegible handwriting, and editorial comments found in the data. To accommodate these problems, a number of arbitrary rules were devised at the data entry stage.
  
  
• In order to compare the most popular term against the indexing record for each shot, three files were created. For each of the 11 shots in the computer database, the indexing terms assigned were listed, including indexing for aspects other than strictly subject indexing, such as time periods, geographic descriptors, and camera angles, which in the database are expressed using classificatory or other structures and represented in lookup tables. For the 33 shots in the card files, the records were photocopied. The recto of these records includes a running description covering the action in entire reel of film, as well as shelf location and other identifying information. The verso includes descriptors assigned to the reel and used to produce card sets for subject access. Finally, the list of most popular terms identified for each shot was divided into two sections, one for the shots recorded in the database and one for the shots with card records.
  
  
• In the first part of the study, a match was sought between any of the most popular terms for each shot and any of the indexing terms assigned for the shot. In the second part of the study, a match was sought between any of the most popular terms for each shot and any word in the running description for the shot. The ratio of successful matches was then calculated.

• The most popular terms for each shot used in the original study are given in table 1. In cases where more than one term achieved the highest score, all the competing terms are given. They are presented here as stemmed for the purposes of the original study, while the terms given in the columns labeled “Indexing term assigned” and “Appeared in description as” are given as they appear in the source collection records.
  
  
• It is noteworthy that generally the shots converted from the card file to the computer database are indexed from the written description, not from the image. On the other hand, in the study the participants saw the image but not the written description. The indexing terms shown are not exhaustive. In some cases several other terms are assigned, but are not necessarily relevant to the 10second segment of the shot seen by participants in the study, or are not given here because they are not related in any way to the most popular term from the study. The absence of any word representing a concept related to the most popular term supplied by participants in the original study is shown in the table as an em-dash (---).

  Table 1

  Most popular terms for each shot

  Shot number	Most popular term	Indexing term assigned	Appears in description as
  1. Computer database
  04	radio	radio	radio
  06	construct, prefab	construction	prefabricated
  07	chicken	chikens	chikens
  11	prairie, field	landscapes
  17	winter, snow	winter [time period descriptor]	winter
  18	lake, aerial, geography	aerial [camera angle descriptor]	aerial
  19	lake, island	landscapes, rivers
  21	stadium	stadiums	stadium
  24	ship, ferry, ocean, boat	ferries	ferry 06 construct, prefab construction prefabricated
  39	cat	cats	cat
  42	airplane, plane, seaplane, transport	seaplanes	seaplane
  2. Card file
  01	flag	ceremonies-NATO forces	flag
  02	child, family, Inuit	Eskimo1
  03	bird	birds	birds2
  05	bird	birds
  08	fish, fisherman	fishing, fishermen	fish, fisherman2
  09	mountain, rock	mountains	mountain
  10	ship, box, man, cargo, open, work	ships	ship, men, work
  12	shop, street, window, store, people	streets, store	shop, shoppers, street, windows
  13	train, forest, mountain	train, mountains	train, mountains
  14	grocery, cash	cash	cash
  15	bird	birds
  16	train, freight	trains, freight	train, freight
  20	winter, snow	winter	winter
  22	buffalo	buffaloes	buffaloes
  23	work, man, fish, fisherman, net	fishing	fishermen, nets
  25	rock, mountain	mountain
  26	woodpecker, bird	birds	woodpecker
  27	old, log	log	log
  28	building, urban	buildings	buildings
  29	bird	birds
  30	university	universities	university
  31	duck, bird	ducks, birds
  32	fireman, firefighter	firemen	firemen
  33	moose, forest	[indexing not available]3	moose, forest
  34	mountain	[indexing not available]4	mountain
  35	sun	sunrises	sunrise
  36	ski	ski	skiers
  37	factory, bottle	dairies, food
  38	computer	computers	computer
  40	deer	deer	deer
  41	native, banquet, Indian, meeting, senior, people, gift, old, elderly	banquets, Indian
  43	cow, cattle, farm	farms	cattle
  44	queen	royalty	queen

  
  
• As regards the analysis related to indexing terms, how successful users would be be in retrieving the shots in this sample is of course a function of the match between their inputs and the indexing terms assigned. For the shots in the database, a match would occur in 9 cases out of 11, or 82% of the time. For the shots in the card file, a match would occur in 28 cases out of 31 (two shots were eliminated from this analysis because the indexing for them was not available), or 90% of the time.
  
  
• As regards the analysis related to the running descriptions of the action in the shots, how successful users would be is a function of the match between their inputs and the presence of that term in the running description. For the shots in the database, a match would occur in 9 cases out of 11, or 82% of the time. Interestingly, the two shots for which a match does not occur are the same two as with the comparison with indexing terms. For the shots in the card file, a match would occur in 25 cases out of 33, or 76% of the time. A summary of these results is given in table 2.

  Table 2

  Summary of results

  	Shots in the database	%	Shots in the card file	%
  Term appears in indexing	9/11	82	28/31	90
  Term appears in description	9/11	82	25/33	76

  
  
• The success rate is encouraging and shows that at least as far as concerns the test database, there is a high rate of indexer-requester consistency (Fidel 1994, 574, citing Cooper 1969, 270). However, two factors which foster the high number of matches need to be explained.

• The shots used in the original research were chosen for the levels of agreement they engendered among indexers in a categorization exercise of 200 shots compiled from two separate random samples from the database and the card file in the source collection. The indexers were given the shot descriptions and a list of 13 categories (12 subject categories and “Other”) and asked to place each shot in a single category. The 51 shots which all 11 participants in the exercise placed in the same categories was used as the pool from which the 44 shots which appeared on the research tapes were taken. Thus there is some likelihood that the high degree of successful matches found in the present study has to do with a kind of builtin subject unity inherent in the choice of shots.
  
  
• Secondly, since the data (consisting of terms given to describe stock footage) showed a strong bias in favour of pre-iconographical or ofness terms, more success in matches with the indexing in the source database can be expected than if the research had been conducted with art images. Databases of art images are almost invariably indexed only at the iconographical or aboutness level of description. Shatford Layne (1994, 585) wonders whether there might be more agreement on the principal and more objective aspects of the subject of an image, and less on the secondary and “subjective” aspects. The high level of agreement between user and indexer in the results obtained here suggest that this is the case. Milstead's (1994, 578) observation that “the known high level of inconsistency in indexing must arise at least partly from differences in perception of the 'aboutness' of the document” does not apply to databases of moving image documents indexed at the shot level, because individual shots are not about anything until they are used in some production which gives them context. To complicate matters, often it is impossible to distinguish between ofness and aboutness for purposes of indexing (Turner 1994, 14-16).
  
  
• The results obtained in this study provoke the gnawing question: Since there is a high rate of matches between user inputs and their occurrence in the running descriptions, do we need indexers? The data suggest that indexers name the obvious, as did participants in the study. However, the match represents only a single avenue into the data. Indexers provide access points into less obvious aspects of a shot which may be of interest to another user. In addition, they provide tools which help the user sort out the vocabulary by showing semantic relationships between terms, suggesting related material which might be of use. It has been amply demonstrated that much of the information useful for retrieving an image is not available from the image itself. In a database that takes many facets into account, the important information for some requests may not be available from the running description either.
  
  
• The difficulties of subject indexing in general are numerous and complicated, and it is doubtful whether they can ever be resolved satisfactorily. However, our question about the need for indexers becomes more pertinent when targeted more precisely to the subject indexing of non-art images. The results of the present study strongly suggest that automating subject access to non-art images indexed at the shot level can be successfully achieved. While the level of matches achieved is high in the present results, no special conditions favoring this were present in the creation of the source data. If in future those who select the material and write the running descriptions are made aware that they are also indexing the shot for later retrieval, they can arrange to include in the descriptions any terms thought useful for retrieval. Along with other techniques such as browsing (O'Connor 1993) and “enriched access tools” (O'Connor 1994), indexing derived from running descriptions may prove to be an important member of a team of useful retrieval strategies available to the user.

• The results of the present study strongly suggest that the question of using running descriptions for moving images to derive subject indexing for those images should be further investigated. Moving image databases for stockshot and archival purposes need to include descriptions of the action in each shot, and these shotlevel descriptions are invaluable to the user of a moving-image database since they are an efficient way to reduce retrieval sets to a manageable shot list for viewing (Turner 1990, 6). The results of the study apply to non-art moving images but not to “ordinary” (i.e. non-art) still images, since these latter are usually not accompanied by running descriptions.
  
  
• Nor do the results apply to collections of art images, still or moving, but they suggest that indexing collections of art images at the pre-iconographic ( ofness ) level in addition to the iconographic ( aboutness ) level would help improve retrieval rates, at least for some users. Successfully retrieving art images from information systems is probably more dependent on access points such as the name of an author or a title, but being able to name persons, objects, and events that occur in art pictures would make possible search questions that cannot be asked of a system without this kind of indexing. As is the case in other areas of study, it has become evident that rather than discussing which strategy is better than the others, we should be investigating which combination of strategies yields the best results for various purposes.

• This work was completed with the financial assistance of the Comité d'attribution des fonds internes de recherche (CAFIR) of the Université de Montréal, and with the help of Johanne Lacroix, who served as research assistant for the study.

• 1-No longer considered politically correct, the derogatory term Eskimo is replaced by the term Inuit in the online database. In the present case, the term Eskimo appears in the indexing but not in the part of the description related to the shot. When the record for this footage is included in the online database Eskimo will be changed for Inuit in the indexing. In that sense a match with the most popular term Inuit occurred here.
  
  
• 2-In 1988 a new policy was adopted in the stockshot library. The card file, begun in the early 1940s, recorded information in English only; after 1988 descriptions for shots from French-language productions were in French, and those from English-language productions in English. The indexing remained in English for all shots. In 1991 the policy changed again, and now all the database records have both descriptions and indexing in both languages. The card records for these shots being caught in the interim policy, the actual term in the description is in French while the indexing descriptor is in English.
  
  
• 3-The card in the file is a copy and the indexing, normally on the verso, does not appear.
  
  
• 4-The card for this shot has been misplaced since the description was copied for the study. Thus the appearance of the term in the description is known from other records of it, but the indexing could not be found.

Cooper, W.S. 1969. Is interindexer consistency a hobgoblin? American Documentation 20, 268-78.

Fidel, Raya. 1994. User-centered indexing. Journal of the American Society for Information Science 45, no. 8, 572-76.

Furnas, G.W., T.K. Landauer, L.M. Gomez, and S.T. Dumais. 1983. Statistical semantics: analysis of the potential performance of keyword information systems. The Bell System Technical Journal 62, no. 6 (July-August): 1753-1806.

Milstead, Jessica L. 1994. Needs for research in indexing. Journal of the American Society for Information Science 45, no. 8, 577-82.

O'Connor, Brian. 1993. Browsing: a framework for seeking functional information. Knowledge 15, no. 2, 211-31.

O'Connor, Brian. 1994. Concepts and techniques for use-specific representations of documents. Libres: Library and Information Science Research Electronic Journal 4, no. 4 (December 31).

Shatford Layne, Sara. 1994. Some issues in the indexing of images. Journal of the American Society for Information Science 45, no. 8, 583-8.

Turner, James. 1990. Representing and accessing information in the stockshot database at the National Film Board of Canada. Canadian Journal of Information Science 15, no. 4 (December), 1-22.

Turner, James Ian Marc. 1994. Determining the subject content of still and moving image documents for storage and retrieval: an experimental investigation. PhD thesis, University of Toronto.

Main menu | EBSI | Comments

Other presentation