Mandatory numeracy tests have become commonplace in many countries, heralding a new era in school assessment. New forms of accountability and an increased emphasis on national and international standards (and benchmarks) have the potential to reshape mathematics curricula. It is noteworthy that the mathematics items used in these tests are rich in graphics. Many of the items, for example, require students to have an understanding of information graphics (for example, maps, charts and graphs) in order to solve the tasks. This investigation classifies mathematics items in Australia's inaugural national numeracy tests and considers the effect such standardised testing will have on practice. It is argued that the design of mathematics items is more likely to be a reliable indication of student performance if graphical, linguistic and contextual components are considered both in isolation and in integrated ways as essential elements of task design.