How good are our universities?

We have 15 universities proclaimed to be institutions of higher learning by relevant Acts of Parliament, but, are they institutions of higher learning in practice?

The need for creative knowledge for a knowledge economy in Sri Lanka is endorsed by many and our universities are exhorted to lead the effort, but, exactly how we do that is yet to be explored. 

The World Bank has funded our universities in the last 15 years or more, first, under the ‘Improving the quality and relevance of university education (IRQUE)’ project for $40.3 million, and then under the ‘Higher education for the twenty first century’ project for $ 40 million. Have these efforts made a qualitative difference in our universities? In the absence of a systematic collection and dissemination of performance measures, we really don’t know.

In fact, in my opinion, the bank’s money would have been better spent if its efforts were simply directed towards a sustainable system for quality assurance and performance evaluation of our universities. 

Arthur Stinchcombe in his 1990 book ‘Information and organisations’ has a chapter on universities as organisations for which he uses the sub title ‘Managers who do not know what their workers are doing’. Higher education presents exactly the same dilemma to ministers, secretaries of ministries and other who try to get a grip on higher education. University teachers hire other teachers and award each other professorships in processes that are not public or understandable by the public. Are they propagating mediocrity? Currently we really have no way of assessing. 

During the 1999-2000 period I had the opportunity work with a small team of planners at the Ohio State University, the largest university in the USA then, to develop an academic score card for the institution. It was developed with 10 other aspiration peers as benchmarks. The score card initiated intense debate within a university, acrimonious often, but, a renewed sense of quality was a result, I think. Within ten years or so the university moved up the ranks among its 10 aspirational peers. 

We have 15 universities proclaimed to be institutions of higher learning by relevant Acts of Parliament, but, are they institutions of higher learning in practice? 

While the professional degrees in medicine and engineering are coveted for the ‘free of charge’ education they provide, ministers, MPs, government officials or even university faculty themselves and anybody who can afford to do so choose to send their children abroad for higher education. In the Faculties of Arts which enrol one third of the university intake, the 85% of the intake is female and monks are a highly visible component of the male student population. Young men have voted with their feet to say that the education provided by our faculties of Arts is not adequate. There, you have a living example of a quality measure. 

The market place for higher education suffers from information asymmetry. Those who provide education are able to do so because they have knowledge credentials. Their outputs are also credentials in the form of certificates. The users of these credentials, employers and the tax payers, parents and students who paid for the education have no way of judging its quality. 

There are two main imperfect, but, practical ways that students and their parents receive information about standards and quality in higher education. 

• Quality assurance – to establish baseline standards on inputs and processes
• Rankings or comparative evaluation of performance – on outputs and outcomes and some inputs and outputs 

In quality assurance processes, an independent entity will set academic standards and ensure that institutions adhere to those standards, and that they have all the systems, resources and information necessary for maintaining and improving standards and quality. 

In ranking surveys or league tables, further information for students and parents is provided to compare between institutions. Although ranking methodologies have been criticised for their shortcomings, they have become an essential part of the higher education sector in any country. 

If quality assurance is the cake, ranking is the icing. 

Before we look at each in detail we will take a brief tour of the higher education landscape in Sri Lanka. 

1. Higher education landscape in Sri Lanka

In 2011, LIRNEasia carried out a survey of institutions awarding degrees or degree-equivalent professional qualifications such as CIMA passed-finalist qualifications. We found that public universities and private universities together awarded 19,599 degrees in the 2010/2011 period with public universities with admission through UGC, public institutions with open admission and private institutions accounting for 64%, 22% and 14% respectively, of the degrees awarded. We estimated professional bodies to award 1,000 or more degree-equivalent qualifications. The results were disseminated at a public event hosted by the Ceylon Chamber of Commerce.

In a follow up survey of information available in the public domain we found 600+ education and training programs offered by 200+ plus institutions in 30+ sectors from accountancy, architecture, and aviation to logistics, to teaching and tourism. The results were published in the first-half of 2014 Education times and ‘Ada’ newspaper. Some of the information is posted on the Education Forum website as a career guide.

Even with a preponderance of marketing by private institutions the set of 15 public universities is still the major avenue of further education for youth in Sri Lanka, the rural youth, in particular. 

2. Quality assurance

A Quality Assurance and Accreditation Council (QAAC) of Sri Lanka was established in 2003 with the support of World Bank’s IRQUE project. With the previous Ministry unhappy with the results perhaps, QAAC role has essentially disappeared. The process seems to have been revitalised in a less ambitious way through the Standing Committee on Quality Assurance at University Grants Commission with universities advised to do their own in-house quality assurance.

A quality assurance process typically consists of an internal evaluation and external review of institution and the academic programs. The QAAC functioned under the UGC and facilitated the reviews. For example, program review was carried out along seven dimensions – Curriculum Design, Content and Revie; Teaching, Learning and Assessment Methods; Quality of Students; The Extent and Use of Student Feedback, Qualitative and Quantitative; Postgraduate Studies; Peer Observations and Skills Development.

All sound well and good on paper, but, the process was essentially having a set of faculty members form the 15 universities going around and evaluating each other. The results were published, but, not available anymore on the Web. No big harm there, in my opinion, because the QAAC results are what we call a ‘washout’, i.e. shows no discernible differences across programs. 

In fairness to the QAAC, quality assurance processes are not meant for comparing programs, but, for ensuring that they adhere to certain basic standards. In Sri Lanka, the problem is the small size of the pool of universities. Unless we incur the cost of getting a few external evaluators from top ranked universities in USA, UK etc., within country QA processes will not be credible.

3. Rankings or comparative evaluations of performance

In USA, UK, Canada and Australia, systems of national league tables or national rankings have evolved over time. Increasingly, universities outside of USA and Europe are becoming players in the international arena, with China having adopted a national policy of emerging as a leader in higher education in the world. The ranking survey by the Shanghai Jiao Tong University (SJTU) covering 500 top universities in the world signifies this shift. The Times Higher Education Supplement (THES) of the UK annually publishes the ‘Top 200 universities in the world’ survey adding some competition to international rankings. The survey by Asia Week filled the need for regional survey for Asia but unfortunately that survey ended in 2000 with the demise of the magazine. The Webometrics ranking, based on information available on websites of universities, is superficial but it is the most comprehensive international ranking system available.

Information used in league tables includes input data such as quality of faculty and the quality of the incoming students and some process data such class size. A third category of information used in such rankings is a reputation score received by each institution from its peers. Finally, an algorithm is used to calculate composite score and the score are ordered in descending order to produce a ranking of institutions. 

Let us look two of the quality indicators in such ranking – faculty quality and teaching effectiveness – in more detail.

Faculty quality

As A. J. Scott, the first Principal of Owens College, Manchester, remarked famously in 1851: “He who learns from one occupied in learning, drinks of a running stream. He who learns from one who has learned all he is to teach, drinks the green mantle of the stagnant pool” (Scott, 1851).

The meaning of “occupied in learning” would vary with the mission of the higher education institution. For institutions dedicated to undergraduate education, occupied in learning would mean keeping abreast of developments in their field and continually updating their teaching tools and resources. In some of the highly rated liberal arts colleges in the USA, faculty are not judged by the academic publications they produce, but, the quality of their undergraduate teaching curriculum and delivery. 

The idea of different ways of engaging in learning was first formalised by Ernest Boyer in 1980 in his seminal book titled “Scholarship reconsidered: Priorities of the professoriate”. There he introduced four types of scholarship –scholarship of discovery, scholarship of synthesis, scholarship of application and scholarship of teaching. 

In 2008-2009, as a consultant to the University Grants commission, with late Prof. Senaka Bandaranayake’s guidance, I developed a faculty quality ranking system. It included measures of qualifications, rank and publications. For example in the qualifications category, we gave scores from 1-10 for post-graduate qualifications for the teaching staff in each of academic programs in humanities and social science in the university system. A score of 1 was given for a master’s degree from the same faculty where you received your undergraduate degree, to underscore inadequacy of such a qualification. A score of 10 was given to a PhD from a university in USA, UK or Australia. These quality measures were meant for internal use to identify gaps in faculty resources and correct. We should have weighted the scores according to the mission of each university, but, in an environment where pretenses in equality outweigh efficiency and effectiveness criteria we did venture in that direction. 

Teaching effectiveness

In retrospect, a major problem with the faculty quality system we developed is the absence of measures of scholarship other than the scholarship of discovery, particularly the scholarship of teaching. Faculty may have the basic qualifications, hold rank of associate professor or professor and produce publications, but, are the students getting the benefits?

Even in the USA where universities are often held up as models for international emulation, there is much dissatisfaction with the current state of teaching effectiveness. Students may complete a degree and may even find employment but are the awarded degrees worth the investment? What have the students actually learned? Therefore, higher education interest groups are now experimenting with indicators that directly measure student learning. 

One such is the measure of value addition, where students are evaluated to see what gains they have made in their critical thinking, analytical reasoning, problem solving and communication skills as a result of their education. College Learning Assessment (CAL) instrument developed by Rand Corporation in partnership with the Council for Aid to Education is a case in point. As the developers caution, value addition measures require careful design of the data collection instruments and the rigorous analysis of data to correct for other variables that can affect student learning. In other words, value addition measures can be costly. 

For Sri Lanka, developing value addition outcome measure would be costly. As a consultant to the HETC project, I proposed an alternative method based on student portfolios. We did some pilot work with the Arts Faculties, but, further development of the concept needs approval at the next stage of funding.