Reading CHAT Data#

PyLangAcq is designed to handle conversational data represented in the CHAT format as used in the CHILDES database for language acquisition research; CHAT is documented in its official manual. This page describes the ways CHAT data can be read by the pylangacq package.

Caution

CHAT data is processed in parallelized code to speed things up by default. Especially for Windows users, you may need to put your code under the if __name__ == "__main__": idiom in a script to avoid running into an error. For reference, please see the “safe importing of main module” section for parallelization from the official Python documentation.

Initializing a Reader#

read_chat()#

Reading CHAT data in PyLangAcq is all about creating a Reader object. The most convenient way to do it is to use the read_chat() function, which asks for a data source and several optional arguments. As an example, let’s use the Brown dataset of American English on CHILDES:

>>> import pylangacq
>>> url = "https://childes.talkbank.org/data/Eng-NA/Brown.zip"
>>> brown = pylangacq.read_chat(url)

If your data source is a URL pointing to a ZIP archive file, like the Brown example here or many others from CHILDES, read_chat() automatically handles everything behind the scenes for you, from downloading the ZIP file, unzipping it, traversing through the CHAT files found, as well as parsing the files. If the ZIP file has a fair amount of data (the Brown dataset has over 200 CHAT data files, with over 180,000 utterances), a read_chat() call like this typically takes a couple seconds.

>>> brown.info()
214 files
184631 utterances
880748 words
      Utterance Count    Word Count  File Path
--  -----------------  ------------  ---------------------
#1               1737          6437  Brown/Adam/020304.cha
#2               1972          7757  Brown/Adam/020318.cha
#3               1305          5572  Brown/Adam/020403.cha
#4               1224          4570  Brown/Adam/020415.cha
#5               1344          5469  Brown/Adam/020430.cha
...
(set `verbose` to True for all the files)

For a quick preview of what the data looks like, The head() and tail() methods provide a quick preview of what the data looks like:

>>> brown.head()
*CHI:  play     checkers      .
%mor:  n|play   n|checker-PL  .
%gra:  1|2|MOD  2|0|INCROOT   3|2|PUNCT
%pho:   <1> pe

*CHI:  big      drum         .
%mor:  adj|big  n|drum       .
%gra:  1|2|MOD  2|0|INCROOT  3|2|PUNCT

*MOT:  big      drum         ?
%mor:  adj|big  n|drum       ?
%gra:  1|2|MOD  2|0|INCROOT  3|2|PUNCT

*CHI:  big      drum         .
%mor:  adj|big  n|drum       .
%gra:  1|2|MOD  2|0|INCROOT  3|2|PUNCT
%spa:   $IMIT

*CHI:  big      drum         .
%mor:  adj|big  n|drum       .
%gra:  1|2|MOD  2|0|INCROOT  3|2|PUNCT
%spa:   $IMIT

In practice, you likely only need a subset of the data at a time, e.g., focusing on a particular child. The Brown dataset contains data for the three children Adam, Eve, and Sarah. Suppose you need Eve’s data only. read_chat() takes the optional argument match which, if specified, matches the file paths and only handles the matching data files. To know what the file paths look like and therefore determine what the match argument should be, either you independently have the unzipped files on your system and see the subdirectory structure, or the brown reader we’ve just created can tell you that via file_paths():

>>> brown.file_paths()
['Brown/Adam/020304.cha',
 'Brown/Adam/020318.cha',
 ...
 'Brown/Eve/010600a.cha',
 'Brown/Eve/010600b.cha',
 ...
 'Brown/Sarah/020305.cha',
 'Brown/Sarah/020307.cha',
 ...
 'Brown/Sarah/050106.cha']

It looks like all and only Eve’s data is inside the subdirectory called "Eve". If we pass "Eve" to match, we should be getting only Eve’s data this time (and the function should run and finish noticeably faster due to the much smaller data amount):

>>> eve = pylangacq.read_chat(url, match="Eve")
>>> eve.n_files()
20
>>> len(eve.utterances())
26920

So far, we’ve seen how read_chat() works with a URL that points to a ZIP file. Other data sources that this function is designed for are:

  1. A ZIP file on your local system:

>>> reader = pylangacq.read_chat("path/to/your/local/data.zip")
  1. A directory (i.e., folder) on your local system, where CHAT data files are found immediately or recursively in subdirectories:

>>> reader = pylangacq.read_chat("path/to/your/local/directory/")
  1. A single CHAT file on your system:

>>> reader = pylangacq.read_chat("path/to/your/local/data.cha")

read_chat() is designed to cover the common use cases of reading in CHAT data. Under the hood, it is a wrapper of several classmethods of Reader, some of which aren’t available from read_chat(). These classmethods are introduced in the following.

From a ZIP File or Local Directory#

Perhaps you don’t want read_chat() to do the guess work of what type of your data source is, or you want more fine-grained control of what counts as CHAT data files or not in your data source. While read_chat() already handles a ZIP archive file and a local directory, the Reader classmethods from_zip() and from_dir() allow more optional arguments for customization. Here’s sample code for using these classmethods in the base case:

>>> reader = pylangacq.Reader.from_zip("path/to/your/local/data.zip")
>>> reader = pylangacq.Reader.from_dir("path/to/your/local/directory/")

If you pass in a remote URL of a ZIP archive ("https://.../some_data.zip") to from_zip(), the downloaded data is cached on disk by default, so that the same data won’t have to be downloaded again for the same URL. Setting the boolean keyword argument use_cached to False would force re-downloading. The helper functions pylangacq.chat.cached_data_info() and pylangacq.chat.remove_cached_data() let you check and remove cached data, respectively.

From Specific CHAT Data Files#

If you’d like to target specific files, the Reader classmethod from_files() takes a list of file paths:

>>> path1 = "path/to/one/data/file.cha"
>>> path2 = "path/to/another/data/file.cha"
>>> reader = pylangacq.Reader.from_files([path1, path2])

From In-Memory Strings#

If your CHAT data comes from in-memory strings, the Reader classmethod from_strs() takes a list of strings, where each string is assumed to conform to the CHAT data format:

>>> # Let's create some minimal CHAT data as a string.
>>> data = "*CHI:\tI want cookie .\n*MOT:\tokay ."
>>>
>>> # We should see two utterances.
>>> print(data)
*CHI:       I want cookie .
*MOT:       okay .
>>>
>>> reader = pylangacq.Reader.from_strs([data])
>>> len(reader.utterances())
2
>>>
>>> # All "file" terminology still applies.
>>> # Each CHAT data string you pass in is treated as one "file".
>>> reader.n_files()
1
>>>
>>> reader.utterances()
[Utterance(participant='CHI',
           tokens=[Token(word='I', pos=None, mor=None, gra=None),
                   Token(word='want', pos=None, mor=None, gra=None),
                   Token(word='cookie', pos=None, mor=None, gra=None),
                   Token(word='.', pos=None, mor=None, gra=None)],
           time_marks=None,
           tiers={'CHI': 'I want cookie .'}),
 Utterance(participant='MOT',
           tokens=[Token(word='okay', pos=None, mor=None, gra=None),
                   Token(word='.', pos=None, mor=None, gra=None)],
           time_marks=None,
           tiers={'MOT': 'okay .'})]

We are getting ahead of ourselves by showing the result of the Reader classmethod utterances(). We are going to drill down to this and many other functions in the upcoming parts of the documentation, but this quick example gives you a glimpse of how PyLangAcq represents CHAT data.

Parallel Processing#

Because a CHILDES / TalkBank dataset usually comes with multiple CHAT data files, it is reasonable to parallelize the process of reading and parsing CHAT data for speed-up. By default, such parallelization is applied. If you would like to turn off parallel processing (e.g., because your application is already parallelized, and further parallelization from within PyLangAcq would create undesirable effects), the boolean argument parallel is available at from_zip(), from_dir(), from_files(), and from_strs(), and you may set it to False .

Creating an Empty Reader#

Calling Reader itself with no arguments initializes an empty reader:

>>> reader = pylangacq.Reader()
>>> reader.n_files()
0

An empty reader is useful when you’d like to start with no data and “grow” the reader by having data added as necessary. The section below discusses how to manipulate data in a reader.

Adding and Removing Data#

A Reader keeps the linear ordering of CHAT data by the ordering of the source data files. CHAT data typically comes as data files that each represent a recording session. There is, therefore, a natural ordering of the files by time, for when the recordings were made. The ordering is also commonly reflected by the way CHAT data files are named. For this reason, if your input data source is a ZIP file or local directory, the resulting reader has the data automatically sorted based on file paths.

With the knowledge that data is ordered by files in a Reader, it is reasonable for a Reader to append or drop data, and to do so from either end for modeling purposes. Think of a CHAT data reader more or less like a deque.

The following Reader methods support adding and removing data from a reader:

append(reader)

Append data from another reader.

append_left(reader)

Left-append data from another reader.

extend(readers)

Extend data from other readers.

extend_left(readers)

Left-extend data from other readers.

pop()

Drop the last data file from the reader and return it as a reader.

pop_left()

Drop the first data file from the reader and return it as a reader.

filter([match, exclude])

Return a new reader filtered by file paths.

clear()

Remove all data from this reader.

Among these methods, filter() creates and returns a new Reader without altering the original one. All the other methods work by mutating the calling Reader in-place.

For convenience, the addition operator + is defined for Reader objects, and can be used to concatenate two Reader objects. By extension, += is also valid, so a statement in the form of reader1 += reader2 would mutate reader1 by concatenating the two readers.

A Reader can be iterated upon (e.g., for reader_one_file in reader: ...), where the element in each iteration is a Reader for one data file. Slicing (reader[:5], reader[3:6], etc) is also supported, which gives you a Reader object (which is iterable) for the specified data files. To inspect what data files are in a reader and their ordering (as well as extract their indices, if necessary), file_paths() gives you the list of file paths.

The following example illustrates how to build a reader of Eve’s utterances starting from an empty one and adding data to it one file at a time.

>>> reader = pylangacq.Reader()  # an empty reader
>>> for eve_one_file in eve[:5]:
...     reader.append(eve_one_file)
...     print(
...         "Number of Eve's utterances in the reader so far:",
...         len(reader.utterances(participants='CHI'))
...     )
...
Number of Eve's utterances in the reader so far: 741
Number of Eve's utterances in the reader so far: 1214
Number of Eve's utterances in the reader so far: 1467
Number of Eve's utterances in the reader so far: 2052
Number of Eve's utterances in the reader so far: 2758

filter() is designed to explicitly return a new Reader so that we can instantiate a source Reader for a CHILDES / TalkBank dataset and filter it down to specific file paths to get smaller Reader objects. Typically, a CHILDES dataset contains multiple children’s data organized by a subdirectory structure. filter() allows us to easily create Reader objects for individual children without re-downloading data:

>>> url = "https://childes.talkbank.org/data/Eng-NA/Brown.zip"
>>> brown = pylangacq.read_chat(url)
>>> len(brown.file_paths())  # All CHAT files in the Brown dataset
214
>>>
>>> # Eve's data is all Brown/Eve/*.cha -- match the "Eve" substring
>>> eve = brown.filter(match="Eve")
>>> len(eve.file_paths())  # Only the 20 CHAT files for Eve
20
>>> eve.file_paths()[:3]
['Brown/Eve/010600a.cha', 'Brown/Eve/010600b.cha', 'Brown/Eve/010700a.cha']
>>>
>>> # Sarah's data is all Brown/Sarah/*.cha -- match the "Sarah" substring
>>> sarah = brown.filter(match="Sarah")
>>> len(sarah.file_paths())  # Only the 139 CHAT files for Sarah
139
>>> sarah.file_paths()[:3]
['Brown/Sarah/020305.cha', 'Brown/Sarah/020307.cha', 'Brown/Sarah/020319.cha']

Custom Behavior#

If custom behavior in CHAT handling is needed, consider defining a child class that inherits from Reader. This approach is suitable if, for instance, new class methods are needed, or the words / tokens / utterances need a custom treatment during CHAT parsing. As long as you have conversational data formatted in CHAT (data not necessarily from CHILDES or TalkBank, and not necessarily for language acquisition research, either), subclassing from Reader is a powerful way to modify and extend the CHAT handling capabilities. As an example, please see the class CHATReader for Cantonese conversational data in the PyCantonese package.