An automatic behavior recognition system classifies animal behaviors using movements and their temporal context

Animalscanperformcomplexandpurposefulbehaviorsbyexecutingsimplermovementsinflexiblesequences. It is particularly challenging to analyze behavior sequences when they are highly variable, as is the case in language production, certain types of birdsong and, as in our experiments, flies grooming. High sequence variabilitynecessitatesrigorousquantificationoflargeamountsofdatatoidentifyorganizationalprinciplesand temporal structure of such behavior. To cope with large amounts of data, and minimize human effort and subjective bias, researchers often use automatic behavior recognition software. Our standard grooming assay involves coating flies in dust and videotaping them as they groom to remove it. The flies move freely and so performthesamemovementsinvariousorientations.Asthedustisremoved,theirappearancechanges.These conditionsmakeitdifficulttorelyonprecisebodyalignmentandanatomicallandmarkssuchaseyesorlegsand thuspresentchallengestoexistingbehaviorclassificationsoftware.Humanobserversusespeed,location,and shapeofthemovementsasthediagnosticfeaturesofparticulargroomingactions.Weappliedthisintuitionto designanewautomaticbehaviorrecognitionsystem(ABRS)basedonspatiotemporalfeaturesinthevideodata, heavilyweightedfortemporaldynamicsandinvarianttotheanimal’spositionandorientationinthescene.We usethesespatiotemporalfeaturesintwostepsofsupervisedclassificationthatreflecttwotime-scalesatwhich thebehaviorisstructured.Asaproofofprinciple,weshowresultsfromquantificationandanalysisofalarge datasetofstimulus-inducedflygroomingbehaviorsthatwouldhavebeendifficulttoassessinasmallerdataset ofhuman-annotatedethograms.Whilewedevelopedandvalidatedthisapproachtoanalyzeflygroomingbehavior,weproposethatthestrategyofcombiningalignment-invariantfeaturesandmulti-timescaleanalysismay begenerallyusefulformovement-basedclassificationofbehaviorfromvideodata.