Catalogue description Ministry of Agriculture, Fisheries and Food: Statistics (Census and Prices) Division: Agricultural and Horticultural Census Datasets

Agricultural Census and Survey System (ACS) (1968-1993):

• Hardware: ICL mainframes.
• Operating System: GEORGE (II & III).
• Application Software: Inhouse system written in COBOL. Enhanced and refined until its replacement by the Farm Survey System (FSS) in 1993. Tabulation Utility (TAU) was used in connection with the EC Structure Survey. In addition, there were 470 job control macros.

Farm Survey System (FSS) (1993-?):

• Hardware: Client/Server architecture operated via a Local Area Network (LAN). The FSS LAN was connected to the MAFF data communications network (MAFFnet). The FSS ran on two servers: FSS1, the main server. This managed the Ingres relational database in which data from censuses and surveys was stored. FSS2, was used by statisticians in Stats(C&S) to carry out complex statistical analyses of the data held on FSS1.
• Operating System: UNIX.
• Application Software: Ingres database management.

User Interface: Not known for ACS. FSS allowed up to 80 users to connect to the main server (FSS1) concurrently. Users moved through the system via a hierarchy of screens and menus.

Logical Structure and Schema: Each of the Parish Summaries datasets consists of a single table of aggregated data for all English and Welsh parishes for a particular census. The Counties and Parishes lookup tables likewise consist of a single table each.

In MAFF's view, the system can be divided into three main sub-systems:

• A fixed application, the same for all censuses and surveys run on the system.
• A survey application specific to each survey.
• Software tools to manipulate and analyse the data.

How data was originally captured and validated: During the 1989-1996 censuses, the basic method of data capture was paper forms. Farmers were asked to return the forms within about a week of the census date in mid June. Non-respondents were sent reminders. MAFF had the power to prosecute non-respondents.

Before 1997, farmers were sent either a 'short' or a 'long' form. The short omitted the sections on horticulture and glasshouses. Farmers were sent a long form unless previous census returns indicated that no horticulture or glasshouse data was likely to be present.

Before 1995 forms were sent to all holdings which were classed as 'main' holdings. In 1995 an element of sampling was introduced. Thereafter, sampling was applied to main holdings with a Standard Gross Margin (SGM) of less than 9600 (SGM was 'an economic measure of a farm's profitability . . . derived from the activity being recorded on the Agricultural Census return for the farm'). 'Small main holdings', were sent a census form every three years on a rolling basis; 'very small main holdings' (those with an SGM of less than 3600) were sampled randomly each year at a rate of 10%.

MAFF also attempted to integrate data collection for the census with data collection for its Integrated Administration and Control System (IACS). By 1996 Stats(C&S) were investigating the possibility of establishing closer links between IACS and the census. By 1999 this has progressed to the point that census data was derived from IACS for about 6,000 farms whose production was covered by an IACS claim.

Data inputting and validation: In the 1989-1996 censuses, the forms were returned by farmers to the Parish List Section of Stats(C&S). The rate of return of forms could be as high as 10,000 in a single day. Under the ACS, receipting was done by batching the forms and sending them to ITD's PCK section for the recording of the holding numbers ('CPHing'). With the introduction of the FSS in 1993, bar-coded forms were introduced. After forms were receipted, they were scrutinised by the Parish List clerks. Data inputting was done by batching the forms and sending them to PCK section for speed keying. This continued to be the normal method of data entry after the introduction of the FSS. However, the Parish List clerks acquired the ability to input the data directly to the FSS via input screens.

Once the forms were loaded into the system, a parameter sheet was generated which listed batches loaded, and the number of forms in each. A data load report (recording forms that failed to load) was also produced, and a validation error report (listing validation queries by county in the order of CPH numbers). The data was validated against previous returns, against other values on the form, and for credibility via validation rules written by statisticians. Validation queries were cleared by Parish List staff by reference to the original survey forms, by following standard instructions, and by contacting the farmer directly.

Provisional results for the census were published after about 60% of the forms had been received. Final results were calculated when about 80% of the despatched forms were deemed sufficiently 'clean' to be used. In order to account for non-responses statistical techniques were employed to generate a dataset covering 100% of main holdings. Two techniques were used:

• Ratio estimation: 'using previous years data as a base . . . the returned sample is poststratified by farm size (SGM) and farm type to minimise non-response bias. Each item being measured is raised independently of others and so different raising factors apply. Coverage and standard errors for the estimates vary between items but relative standard errors are typically under 2 per cent for the main crops and under 5 per cent for the livestock'.
• Imputation: 'this involves applying an appropriate trend (obtained from responding units) to the previous year’s data for a missing farm to give a best estimate of the current year's information'.

Production of final results involved the use of estimation to compensate for holdings which had not returned a form, or for returns which were being queried.

Constraints on the reliability of the data: MAFF stated that the Census was 'subject to some margin of statistical error on account of the quality of the basic material and the methods which are used in adjusting and statistically treating data for non-respondents'.

Validation performed after transfer: Details of the content and transformation validation checks performed by NDAD staff on each Agricultural and Horticultural Census dataset are contained in the catalogues of individual datasets.

The Agricultural and Horticultural census dates back to the nineteenth century. The first agricultural statistics to have complete coverage of Great Britain were collected in 1866 by the Board of Inland Revenue on behalf of the Board of Trade, which was responsible for the processing and publication of the returns. In 1883 publication was undertaken for the first time by the Agricultural Department of the Privy Council, to which, in 1884, the grant made to the Board of Trade for their collections was transferred. The Agricultural Department of the Privy Council began to process the forms for itself although the actual collection remained in the hands of the Inland Revenue. These arrangements continued unchanged in the succeeding years, during which (in 1889) the Agricultural Department of the Privy Council became a part of the newly formed Board of Agriculture. Later, in 1909, the excise duties of Inland Revenue were transferred to the Board of Customs. This Board became responsible for the collection of the returns until 1912, when the duty was transferred to the Board of Agriculture and Fisheries. The Board became the Ministry of Agriculture and Fisheries (MAFF) under the Ministry of Agriculture and Fisheries Act 1919. Responsibility for the census in Scotland was devolved in 1912 to the Board of Agriculture for Scotland (part of the Scottish Office).

The completion of the census returns by farmers was done on a voluntary basis from 1866 until the passing of the Corn Production Act 1917. The Act provided for the payment of a subsidy to farmers based on the acreage of their wheats and oats, and in turn made the completion of the census returns obligatory. Voluntary completion was re-introduced in 1921, when the subsidy paid under the Act was withdrawn. The completion of census returns became compulsory again with the Agricultural Returns Act 1925, which established the statutory authority for the census. Successive enactments since then have laid an obligation on farmers to complete the forms for the Agricultural and Horticultural Census. Since 1956 farmers have returned the forms directly to MAFF's central statistical departments. Before then, forms were collected at a local level by MAFF's County Advisory Officers (1949-1956); by a network of MAFF-employed crop reporters (1919-1949); and by excise officers (1866-1918).

The current statutory basis for the census is the Agricultural Statistics Act 1979, as amended by the Agriculture (Amendment) Act 1984, and Section 12 of the European Communities Act 1972. The legislation requires farmers in England and Wales to provide information for statistical purposes, as required by the Minister of Agriculture, Fisheries and Food and the Secretary of State for Wales. It also requires that strict confidentiality be observed with respect to the data, to protect against the disclosure of information on individual holdings. One effect of this is that data gathered in the Agricultural and Horticultural Census has only been published and made available in aggregated form.

In addition to data for publication, the Agricultural and Horticultural Census provides MAFF with the core data from which sample selections for other surveys are taken. Data from the census is often correlated to data collected in other surveys: e.g. the area of wheat is correlated to the yield. Where there is a correlation, information from the June census may be used to raise the aggregates from the sample in a survey to estimate the aggregates for the population as a whole. Thus the June census is not only important in its own right but also for the processing of many other surveys.

Information derived from the census is provided to MAFF economists and policy sections, and is used in the formulation of agricultural policy, assessment of the impact of European Community policy proposals, identification of market trends, assessment of research and development needs, and in agricultural advisory work. MAFF economists carry out survey work based on data from the June census, a prime example being the Farm Business Survey. Information from the census has also been supplied to agricultural industry customers (such as the Meat and Livestock Commission and the Milk Marketing Board, now succeeded by Milk Marque), and to other government departments: e.g. the Department of the Environment (for research into changes in land use) and the Office for National Statistics (for Social Trends and Regional Trends publications, and for national employment data). Trade associations and non-departmental public bodies produce economic analyses using census data, whilst academics use the material for both research and teaching. Local authorities rely on Agricultural and Horticultural Census data for administration and the provision of statutory services: e.g. structure plans and strategic planning. Results are also used by international bodies. The census and the parallel June censuses carried out in Scotland and Northern Ireland are the main UK sources for the European Community's Farm Structure Survey, a survey conducted by EC member states every two or three years.

Within the Ministry of Agriculture, Fisheries and Food, the responsibility for the Agricultural and Horticultural Census rested (in 2000) with Statistics (Census and Surveys) Division (Stats(C&S)) in the Economics and Statistics Group. Stats(C&S) design the census, gather and analyse the data, produce statistical publications, and maintain the English part of the Parish List (a register of agricultural and horticultural holdings which is used to contact farmers for the census.

Prior to the creation of the National Assembly for Wales in 1999, responsibility for maintaining the Welsh part of the Parish List, and for controlling and correcting Welsh data for the June census and larger surveys, rested with the Statistical Directorate of the Welsh Office and the Directorate's predecessors. Smaller surveys were processed by MAFF on behalf of the Welsh Office. Welsh Office staff therefore had access to MAFF's Farm Survey System, and performed their own data processing, cleaning and validation for the census using the same suite of programs. Although the Welsh Office Agriculture Department appears on some of the Welsh census forms, and as the joint author of some publications relating to the census, NDAD was told that the Department was not directly involved in the processing of census data. The census forms used in Wales from 1989 to 1993 have 'Welsh Office Agriculture Department' at the top of the form, but the contact address is given as Economic and Statistical Services Division, the predecessor of the Welsh Office's Statistical Directorate. In 1999 the Welsh Office Statistical Directorate became the Statistical Directorate of the National Assembly for Wales Executive, to which responsibility for the census in Wales was also transferred.